2024-03-29T11:21:33.350 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:21:33.350 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/orig.config.yaml 2024-03-29T11:21:33.385 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:21:33.472 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622717' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQSUy3yjxAe3hzCysAxj6mi6w9pBA8hIMXdJW134/u3hWyuyWtQiYzvUWmAkLB5se5uJTsiT3NHlNnPnP7UCFE= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvWN01mvFk7+LYeo9d5kf4S4xDRmeshlRoq70u6g/fyxryJ0WCJ9fmGzz1gk3ng0Bf1y7+XgNUOixyAHcUjHUU= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:21:33.473 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:21:33.473 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:21:33.473 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:21:33.475 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:21:33.506 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:21:33.506 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:21:33.506 INFO:teuthology.packaging:ref: None 2024-03-29T11:21:33.506 INFO:teuthology.packaging:tag: None 2024-03-29T11:21:33.506 INFO:teuthology.packaging:branch: main 2024-03-29T11:21:33.506 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:21:33.507 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:21:33.885 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:21:33.887 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:21:33.893 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:21:33.894 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:21:33.905 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:21:33.918 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:21:33.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:21:33.952 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:39.636671', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQSUy3yjxAe3hzCysAxj6mi6w9pBA8hIMXdJW134/u3hWyuyWtQiYzvUWmAkLB5se5uJTsiT3NHlNnPnP7UCFE='} 2024-03-29T11:21:33.975 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:39.637900', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvWN01mvFk7+LYeo9d5kf4S4xDRmeshlRoq70u6g/fyxryJ0WCJ9fmGzz1gk3ng0Bf1y7+XgNUOixyAHcUjHUU='} 2024-03-29T11:21:33.976 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:21:33.987 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:21:33.987 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:21:33.987 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:21:34.064 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc7407510d0>, signals=[15]) 2024-03-29T11:21:34.064 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:21:34.076 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:21:34.076 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:21:34.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:21:34.185 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:21:34.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:21:34.274 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:21:34.286 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2024-03-29T11:21:34.303 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2024-03-29T11:21:34.304 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2024-03-29T11:21:34.360 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="9" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="9" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:21:34.361 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:21:34.362 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:21:34.362 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2024-03-29T11:21:34.381 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-03-29T11:21:34.399 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-03-29T11:21:34.399 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-03-29T11:21:34.456 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-03-29T11:21:34.456 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-03-29T11:21:34.456 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-03-29T11:21:34.456 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:21:34.456 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:21:34.457 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:21:34.458 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-03-29T11:21:34.481 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:21:34.492 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:21:34.504 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:21:34.504 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:21:34.507 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:21:34.523 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:21:34.563 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:21:34.564 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:21:34.566 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:21:34.653 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:21:34.711 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:21:34.730 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:21:34.730 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:21:34.731 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:21:34.731 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:21:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:34.748 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-29T11:21:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:34.749 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-29T11:21:34.805 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:21:34.805 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-361.el9.x86_64 2024-03-29T11:21:34.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-03-29T11:21:34.807 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:21:34.807 INFO:teuthology.task.kernel:Running kernel on smithi022: 5.14.0-361.el9.x86_64 2024-03-29T11:21:34.807 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum install -y kernel 2024-03-29T11:21:36.920 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:21:36.994 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:21:38.404 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-29T11:21:38.603 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-29T11:21:40.369 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-03-29T11:21:40.444 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-03-29T11:21:42.187 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-29T11:21:49.562 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-29T11:21:51.237 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-29T11:21:54.072 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 21 MB 00:02 2024-03-29T11:21:54.160 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-03-29T11:21:58.244 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-29T11:21:59.213 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-29T11:21:59.837 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-29T11:22:00.866 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-29T11:22:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:22:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:22:01.648 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:22:01.659 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:22:01.659 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:22:01.659 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout:Install 4 Packages 2024-03-29T11:22:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:22:01.661 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 76 M 2024-03-29T11:22:01.661 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 123 M 2024-03-29T11:22:01.661 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:22:01.823 INFO:teuthology.orchestra.run.smithi022.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:22:02.648 INFO:teuthology.orchestra.run.smithi022.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 15 MB/s | 15 MB 00:00 2024-03-29T11:22:03.332 INFO:teuthology.orchestra.run.smithi022.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T11:22:03.765 INFO:teuthology.orchestra.run.smithi022.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-29T11:22:03.768 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:03.768 INFO:teuthology.orchestra.run.smithi022.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-29T11:22:03.768 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:22:04.199 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:22:04.199 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:22:05.066 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:22:05.067 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:22:05.938 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-29T11:22:07.507 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:22:08.128 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:08.179 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:22:08.179 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:22:08.216 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:08.267 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:22:08.276 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:22:08.276 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:22:08.276 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:22:08.276 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:22:08.276 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-03-29T11:22:08.277 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:22:08.278 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 76 M 2024-03-29T11:22:08.278 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 123 M 2024-03-29T11:22:08.278 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:22:08.440 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:22:09.090 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-03-29T11:22:09.674 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 22 MB/s | 28 MB 00:01 2024-03-29T11:22:09.982 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T11:22:09.986 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:09.986 INFO:teuthology.orchestra.run.smithi204.stdout:Total 45 MB/s | 76 MB 00:01 2024-03-29T11:22:09.986 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:22:10.363 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:22:10.363 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:22:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:11.245 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:22:11.246 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:22:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:13.527 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:22:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:14.450 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:16.323 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:16.370 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:17.223 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:17.372 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:21.465 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:22.091 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:22.212 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:26.961 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:52.099 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:52.099 INFO:teuthology.orchestra.run.smithi022.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:22:52.099 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:22:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:52.503 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:52.503 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:52.503 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:52.503 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:52.876 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:22:52.877 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:22:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:22:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:22:52.877 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:22:53.215 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:53.324 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:53.325 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:53.325 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:22:53.325 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:22:53.325 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:22:53.326 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:22:53.326 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi022.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:22:53.326 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:53.409 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:53.409 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:53.410 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -qi grub2-tools 2024-03-29T11:22:53.447 INFO:teuthology.orchestra.run.smithi022.stdout:Name : grub2-tools 2024-03-29T11:22:53.447 INFO:teuthology.orchestra.run.smithi022.stdout:Epoch : 1 2024-03-29T11:22:53.447 INFO:teuthology.orchestra.run.smithi022.stdout:Version : 2.06 2024-03-29T11:22:53.447 INFO:teuthology.orchestra.run.smithi022.stdout:Release : 68.el9 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Architecture: x86_64 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Group : Unspecified 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Size : 8264254 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:License : GPLv3+ 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Packager : builder@centos.org 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:Vendor : CentOS 2024-03-29T11:22:53.448 INFO:teuthology.orchestra.run.smithi022.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout:Summary : Support tools for GRUB. 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout:Description : 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout:hardware devices. 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:22:53.449 INFO:teuthology.orchestra.run.smithi022.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:22:53.450 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:22:53.450 INFO:teuthology.task.kernel:Updating grub on smithi022 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:22:53.450 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:22:54.041 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2024-03-29T11:22:55.389 INFO:teuthology.orchestra.run.smithi022.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:22:55.419 INFO:teuthology.orchestra.run.smithi022.stderr:done 2024-03-29T11:22:55.422 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-29T11:22:55.438 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.foN7F6KgdE 2024-03-29T11:22:55.438 DEBUG:teuthology.orchestra.run.smithi022:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.foN7F6KgdE 2024-03-29T11:22:55.509 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.foN7F6KgdE 2024-03-29T11:22:55.650 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.foN7F6KgdE is 6KB 2024-03-29T11:22:55.671 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.foN7F6KgdE 2024-03-29T11:22:55.685 DEBUG:teuthology.orchestra.run.smithi022:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:22:55.762 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:22:55.762 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:22:55.762 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:22:55.762 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:22:55.764 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:22:55.928 DEBUG:teuthology.orchestra.run.smithi022:> sudo shutdown -r now 2024-03-29T11:22:56.606 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:56.606 INFO:teuthology.orchestra.run.smithi204.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:22:56.606 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:22:56.649 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:57.030 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:57.030 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:57.030 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:57.031 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:22:57.425 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:22:57.749 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:57.861 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:57.861 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:57.861 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:22:57.861 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:22:57.861 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:22:57.861 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:22:57.861 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:22:57.861 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:57.942 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:57.942 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:57.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-03-29T11:22:57.976 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-03-29T11:22:57.976 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-03-29T11:22:57.976 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-03-29T11:22:57.976 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 68.el9 2024-03-29T11:22:57.976 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8264254 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:22:57.977 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:22:57.978 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:22:57.978 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:22:57.978 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-03-29T11:22:57.978 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:22:57.978 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:22:57.979 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:22:57.979 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:22:57.979 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:22:58.579 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-03-29T11:22:59.980 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:23:00.037 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-03-29T11:23:00.039 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-29T11:23:00.054 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.qGVg8kFLDN 2024-03-29T11:23:00.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qGVg8kFLDN 2024-03-29T11:23:00.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.qGVg8kFLDN 2024-03-29T11:23:00.257 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.qGVg8kFLDN is 6KB 2024-03-29T11:23:00.277 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.qGVg8kFLDN 2024-03-29T11:23:00.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:23:00.362 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:23:00.362 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:23:00.362 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:23:00.362 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:23:00.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:23:00.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-03-29T11:23:25.956 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:23:25.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T11:23:25.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:23:30.522 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:23:30.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-29T11:23:30.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:23:48.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-29T11:23:57.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-29T11:23:57.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:00.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-29T11:24:12.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-29T11:24:12.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:25.970 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:24:34.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T11:24:34.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:35.398 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-29T11:24:35.805 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T11:24:35.805 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:24:35.806 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-29T11:24:35.819 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:24:35.819 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:24:35.819 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:24:35.819 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:24:36.820 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:24:36.820 DEBUG:teuthology.orchestra.run.smithi022:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:24:36.914 INFO:teuthology.orchestra.run.smithi022.stdout:ttyS1 2024-03-29T11:24:36.948 DEBUG:teuthology.parallel:result is None 2024-03-29T11:25:12.973 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:25:27.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-29T11:25:27.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:28.352 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-29T11:25:28.755 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-29T11:25:28.755 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:25:28.755 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-29T11:25:28.770 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:25:28.771 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:25:28.771 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:25:28.771 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:25:29.772 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:25:29.772 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:25:29.850 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-03-29T11:25:29.886 DEBUG:teuthology.parallel:result is None 2024-03-29T11:25:29.886 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:25:29.893 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:25:29.893 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:25:29.896 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:25:29.914 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:25:29.922 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:25:29.940 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:25:29.940 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:25:29.955 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:25:29.985 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:25:29.993 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:25:29.993 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:25:30.021 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:25:30.060 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:25:30.064 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:25:30.074 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:25:30.078 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:25:30.080 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:25:30.087 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:25:30.087 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:25:30.119 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:25:30.150 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:25:30.159 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:25:30.160 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:25:30.187 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:25:30.205 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:25:30.280 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:25:30.336 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:25:30.416 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:25:30.461 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:25:30.461 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:25:30.528 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:25:30.573 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:25:30.633 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:25:30.674 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:25:30.721 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:25:30.721 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:25:30.791 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2024-03-29T11:25:30.793 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-03-29T11:25:30.840 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:25:30.874 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:25:31.295 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:25:31.303 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:25:31.303 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:25:31.314 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:25:31.324 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2024-03-29T11:25:31.377 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: 2024-03-29T11:25:31.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-03-29T11:25:31.455 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-03-29T11:25:31.457 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:25:31.457 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2024-03-29T11:25:31.487 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2024-03-29T11:25:31.487 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-03-29T11:25:31.510 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-03-29T11:25:31.510 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi022.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:25:31.510 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:25:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:31.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:25:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:31.584 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:25:31.585 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:25:31.629 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:25:31.670 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:25:31.682 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:25:31.686 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:25:31.814 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:25:31.830 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:25:31.831 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi022.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:29:53.242 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-03-29T11:29:53.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T11:29:53.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:29:53.379 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-29T11:29:53.462 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T11:29:53.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-29T11:29:53.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:29:53.554 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-29T11:29:53.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-29T11:29:53.637 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:29:53.648 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:29:53.649 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:29:53.649 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:29:53.652 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:29:53.652 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:29:53.683 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:29:53.701 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:29:53.710 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:29:53.727 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:29:53.731 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2024-03-29T11:29:53.745 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2024-03-29T11:29:53.756 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-03-29T11:29:53.762 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:29:53.769 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-03-29T11:29:53.778 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:29:53.786 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:29:53.802 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:29:53.830 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:29:53.833 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:29:53.833 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-29T11:29:53.833 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:29:53.833 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:29:53.833 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:29:53.833 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:29:53.854 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:29:53.857 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-29T11:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:29:53.858 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:29:53.870 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:29:53.870 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:29:53.870 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:29:53.870 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:29:53.876 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:29:53.876 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:29:53.877 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:29:53.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:29:53.879 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:29:53.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:29:54.024 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:29:54.024 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:29:54.136 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:29:54.136 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:29:54.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:29:54.164 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:29:54.165 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:29:54.233 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:29:54.270 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:29:54.270 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:29:54.314 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:29:54.380 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:29:54.381 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:29:54.381 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:29:54.382 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-03-29T11:29:54.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-03-29T11:29:54.411 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:29:54.411 DEBUG:teuthology.orchestra.run.smithi022:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:29:54.480 DEBUG:teuthology.orchestra.run.smithi022:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:29:54.561 DEBUG:teuthology.orchestra.run.smithi022:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:29:54.622 INFO:teuthology.orchestra.run.smithi204.stdout:74 files removed 2024-03-29T11:29:54.628 INFO:teuthology.orchestra.run.smithi022.stdout:check_obsoletes = 1 2024-03-29T11:29:54.630 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum clean all 2024-03-29T11:29:54.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw 2024-03-29T11:29:54.877 INFO:teuthology.orchestra.run.smithi022.stdout:74 files removed 2024-03-29T11:29:54.905 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw 2024-03-29T11:29:55.463 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 160 kB/s | 82 kB 00:00 2024-03-29T11:29:55.724 INFO:teuthology.orchestra.run.smithi022.stdout:ceph packages for x86_64 159 kB/s | 82 kB 00:00 2024-03-29T11:29:55.955 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-29T11:29:56.291 INFO:teuthology.orchestra.run.smithi022.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-29T11:29:56.339 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-29T11:29:56.742 INFO:teuthology.orchestra.run.smithi022.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-29T11:29:56.759 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:29:57.161 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:29:58.175 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-29T11:29:58.670 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-29T11:30:01.077 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-29T11:30:07.949 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 871 kB/s | 8.0 MB 00:09 2024-03-29T11:30:12.483 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-03-29T11:30:18.429 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-29T11:30:20.102 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-29T11:30:21.878 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T11:30:24.698 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 860 kB/s | 19 MB 00:22 2024-03-29T11:30:26.945 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:30:28.996 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-29T11:30:29.553 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:30:29.567 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:30:29.568 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:30:29.569 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout:Install 30 Packages 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 2 Packages 2024-03-29T11:30:29.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:29.571 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 58 M 2024-03-29T11:30:29.571 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:30:30.305 INFO:teuthology.orchestra.run.smithi204.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 9.3 MB/s | 5.3 MB 00:00 2024-03-29T11:30:30.383 INFO:teuthology.orchestra.run.smithi204.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 323 kB/s | 25 kB 00:00 2024-03-29T11:30:30.489 INFO:teuthology.orchestra.run.smithi204.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.9 MB/s | 742 kB 00:00 2024-03-29T11:30:30.564 INFO:teuthology.orchestra.run.smithi204.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 6.5 MB/s | 498 kB 00:00 2024-03-29T11:30:30.728 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-29T11:30:30.864 INFO:teuthology.orchestra.run.smithi204.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 8.6 MB/s | 9.7 MB 00:01 2024-03-29T11:30:30.939 INFO:teuthology.orchestra.run.smithi204.stdout:(6/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-29T11:30:30.956 INFO:teuthology.orchestra.run.smithi204.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 491 kB/s | 45 kB 00:00 2024-03-29T11:30:30.981 INFO:teuthology.orchestra.run.smithi204.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 3.1 MB/s | 131 kB 00:00 2024-03-29T11:30:31.007 INFO:teuthology.orchestra.run.smithi204.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.1 MB/s | 162 kB 00:00 2024-03-29T11:30:31.032 INFO:teuthology.orchestra.run.smithi204.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 6.2 MB/s | 320 kB 00:00 2024-03-29T11:30:31.058 INFO:teuthology.orchestra.run.smithi204.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.8 MB/s | 298 kB 00:00 2024-03-29T11:30:31.083 INFO:teuthology.orchestra.run.smithi204.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T11:30:31.200 INFO:teuthology.orchestra.run.smithi204.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-29T11:30:31.225 INFO:teuthology.orchestra.run.smithi204.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 633 kB/s | 106 kB 00:00 2024-03-29T11:30:31.267 INFO:teuthology.orchestra.run.smithi204.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 661 kB/s | 44 kB 00:00 2024-03-29T11:30:31.326 INFO:teuthology.orchestra.run.smithi204.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-29T11:30:31.351 INFO:teuthology.orchestra.run.smithi204.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-29T11:30:31.376 INFO:teuthology.orchestra.run.smithi204.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-29T11:30:31.402 INFO:teuthology.orchestra.run.smithi204.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:30:31.427 INFO:teuthology.orchestra.run.smithi204.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T11:30:31.453 INFO:teuthology.orchestra.run.smithi204.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 919 kB/s | 46 kB 00:00 2024-03-29T11:30:31.694 INFO:teuthology.orchestra.run.smithi204.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-29T11:30:31.745 INFO:teuthology.orchestra.run.smithi204.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 518 kB/s | 26 kB 00:00 2024-03-29T11:30:31.812 INFO:teuthology.orchestra.run.smithi204.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-03-29T11:30:31.879 INFO:teuthology.orchestra.run.smithi204.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-29T11:30:31.904 INFO:teuthology.orchestra.run.smithi204.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 731 kB/s | 67 kB 00:00 2024-03-29T11:30:31.946 INFO:teuthology.orchestra.run.smithi204.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-29T11:30:31.973 INFO:teuthology.orchestra.run.smithi204.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T11:30:32.030 INFO:teuthology.orchestra.run.smithi204.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-29T11:30:32.434 INFO:teuthology.orchestra.run.smithi204.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 7.3 MB/s | 3.4 MB 00:00 2024-03-29T11:30:32.598 INFO:teuthology.orchestra.run.smithi204.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 5.5 MB/s | 3.1 MB 00:00 2024-03-29T11:30:34.056 INFO:teuthology.orchestra.run.smithi204.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 4.6 MB/s | 20 MB 00:04 2024-03-29T11:30:34.059 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:34.059 INFO:teuthology.orchestra.run.smithi204.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T11:30:34.060 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:30:34.096 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:30:34.096 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:30:34.578 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:30:34.578 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:30:35.045 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 21 MB 00:04 2024-03-29T11:30:36.057 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:30:36.136 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:30:36.177 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:30:36.203 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:30:36.225 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:30:36.242 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:30:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:30:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:30:36.870 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:30:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:30:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:30:37.125 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:30:37.147 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:30:37.193 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:30:37.214 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:30:37.249 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:30:37.317 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:30:37.342 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:30:37.388 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:30:37.424 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:30:37.531 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:30:37.574 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:30:37.615 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:30:37.648 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:30:37.668 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:30:37.701 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:30:38.010 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:30:38.039 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:30:38.130 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:30:38.161 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:30:38.201 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:30:38.651 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:30:38.676 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:30:38.747 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:30:38.823 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:30:38.853 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:30:40.171 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:30:40.968 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:30:41.017 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:30:41.563 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:30:41.595 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:30:41.759 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:30:41.759 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:30:41.760 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:30:41.760 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:41.796 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:30:42.905 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:30:42.924 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:30:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:30:42.924 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:30:42.924 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:30:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:30:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:30:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse 2024-03-29T11:30:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:30:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:30:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:30:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:30:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:30:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:30:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:30:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout:Installing weak dependencies: 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout:Install 30 Packages 2024-03-29T11:30:42.928 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 2 Packages 2024-03-29T11:30:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:30:42.929 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 58 M 2024-03-29T11:30:42.929 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:30:44.871 INFO:teuthology.orchestra.run.smithi022.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.2 MB/s | 5.3 MB 00:01 2024-03-29T11:30:44.955 INFO:teuthology.orchestra.run.smithi022.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 302 kB/s | 25 kB 00:00 2024-03-29T11:30:45.122 INFO:teuthology.orchestra.run.smithi022.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.3 MB/s | 742 kB 00:00 2024-03-29T11:30:45.239 INFO:teuthology.orchestra.run.smithi022.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.2 MB/s | 498 kB 00:00 2024-03-29T11:30:46.030 INFO:teuthology.orchestra.run.smithi022.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 6.4 MB/s | 5.1 MB 00:00 2024-03-29T11:30:46.131 INFO:teuthology.orchestra.run.smithi022.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-29T11:30:46.256 INFO:teuthology.orchestra.run.smithi022.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.0 MB/s | 131 kB 00:00 2024-03-29T11:30:46.356 INFO:teuthology.orchestra.run.smithi022.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.6 MB/s | 162 kB 00:00 2024-03-29T11:30:46.498 INFO:teuthology.orchestra.run.smithi022.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 2.2 MB/s | 320 kB 00:00 2024-03-29T11:30:46.657 INFO:teuthology.orchestra.run.smithi022.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.8 MB/s | 298 kB 00:00 2024-03-29T11:30:46.782 INFO:teuthology.orchestra.run.smithi022.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 795 kB/s | 99 kB 00:00 2024-03-29T11:30:46.983 INFO:teuthology.orchestra.run.smithi022.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 529 kB/s | 106 kB 00:00 2024-03-29T11:30:47.075 INFO:teuthology.orchestra.run.smithi022.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-03-29T11:30:47.108 INFO:teuthology.orchestra.run.smithi022.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:30:47.225 INFO:teuthology.orchestra.run.smithi022.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2024-03-29T11:30:47.284 INFO:teuthology.orchestra.run.smithi022.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-29T11:30:47.326 INFO:teuthology.orchestra.run.smithi022.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 987 kB/s | 41 kB 00:00 2024-03-29T11:30:47.376 INFO:teuthology.orchestra.run.smithi022.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:30:47.434 INFO:teuthology.orchestra.run.smithi022.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 598 kB/s | 35 kB 00:00 2024-03-29T11:30:47.476 INFO:teuthology.orchestra.run.smithi022.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-29T11:30:47.943 INFO:teuthology.orchestra.run.smithi022.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 662 kB/s | 309 kB 00:00 2024-03-29T11:30:48.310 INFO:teuthology.orchestra.run.smithi022.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-29T11:30:48.402 INFO:teuthology.orchestra.run.smithi022.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-03-29T11:30:48.494 INFO:teuthology.orchestra.run.smithi022.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-03-29T11:30:48.594 INFO:teuthology.orchestra.run.smithi022.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-29T11:30:48.703 INFO:teuthology.orchestra.run.smithi022.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-03-29T11:30:48.795 INFO:teuthology.orchestra.run.smithi022.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-29T11:30:49.495 INFO:teuthology.orchestra.run.smithi022.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 2.3 MB/s | 1.6 MB 00:00 2024-03-29T11:30:50.087 INFO:teuthology.orchestra.run.smithi022.stdout:(29/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.4 MB/s | 9.7 MB 00:06 2024-03-29T11:30:50.245 INFO:teuthology.orchestra.run.smithi022.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.5 MB/s | 3.4 MB 00:00 2024-03-29T11:30:51.479 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:30:51.479 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-03-29T11:30:51.479 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-03-29T11:30:51.479 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-03-29T11:30:51.480 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-03-29T11:30:51.480 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-03-29T11:30:51.480 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-03-29T11:30:51.480 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-03-29T11:30:51.481 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:51.862 INFO:teuthology.orchestra.run.smithi022.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 3.1 MB 00:01 2024-03-29T11:30:52.401 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:30:52.425 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:30:52.426 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:52.426 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:30:52.426 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:30:52.426 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:30:52.426 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:52.427 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:30:52.495 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:30:52.495 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:30:53.703 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:30:53.703 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:30:53.703 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:30:53.703 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:30:53.703 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:30:53.703 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:30:53.703 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:30:53.703 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:30:53.704 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:30:53.705 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:30:53.705 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:30:53.705 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:30:53.705 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:30:53.706 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:30:53.706 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:30:53.706 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:30:53.706 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:30:53.706 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:30:53.706 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:30:53.707 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:30:53.707 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.799 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:30:54.800 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:54.801 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:30:55.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test 2024-03-29T11:30:55.719 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:30:55.879 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:55.880 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:30:55.881 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:30:55.881 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-03-29T11:30:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:30:55.881 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 48 M 2024-03-29T11:30:55.881 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 204 M 2024-03-29T11:30:55.881 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:30:56.163 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-29T11:30:56.221 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T11:30:56.255 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:30:56.280 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 506 kB/s | 163 kB 00:00 2024-03-29T11:30:56.839 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 320 kB/s | 187 kB 00:00 2024-03-29T11:30:59.129 INFO:teuthology.orchestra.run.smithi022.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 20 MB 00:15 2024-03-29T11:30:59.132 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:59.132 INFO:teuthology.orchestra.run.smithi022.stdout:Total 3.6 MB/s | 58 MB 00:16 2024-03-29T11:30:59.133 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:30:59.175 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:30:59.175 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:30:59.356 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 14 MB/s | 47 MB 00:03 2024-03-29T11:30:59.359 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:59.359 INFO:teuthology.orchestra.run.smithi204.stdout:Total 14 MB/s | 48 MB 00:03 2024-03-29T11:30:59.373 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:30:59.394 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:30:59.394 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:30:59.663 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:30:59.663 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:30:59.780 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:30:59.780 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:31:00.355 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:31:00.424 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:31:00.459 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:31:00.491 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:31:00.531 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:31:00.559 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:31:01.167 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:31:01.242 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:01.307 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:01.331 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:01.581 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:02.144 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:02.887 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:02.944 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:02.981 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:03.014 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:03.377 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:04.024 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:04.054 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:04.158 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:04.230 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:04.257 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:31:05.134 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:31:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:07.052 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:07.089 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:07.178 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:07.178 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:07.179 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:07.179 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:31:07.207 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:07.635 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:31:07.635 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:31:07.635 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:31:07.636 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:31:07.636 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:31:07.636 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:08.049 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:31:08.340 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph 2024-03-29T11:31:08.924 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:31:09.098 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:31:09.101 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:31:09.102 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:31:09.103 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout:Install 36 Packages 2024-03-29T11:31:09.104 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:09.105 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 30 M 2024-03-29T11:31:09.105 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 107 M 2024-03-29T11:31:09.105 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:31:09.802 INFO:teuthology.orchestra.run.smithi204.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-03-29T11:31:10.785 INFO:teuthology.orchestra.run.smithi204.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 4.8 MB/s | 4.7 MB 00:00 2024-03-29T11:31:10.902 INFO:teuthology.orchestra.run.smithi204.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-29T11:31:11.111 INFO:teuthology.orchestra.run.smithi204.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 2.1 MB 00:01 2024-03-29T11:31:11.136 INFO:teuthology.orchestra.run.smithi204.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.0 MB/s | 247 kB 00:00 2024-03-29T11:31:11.261 INFO:teuthology.orchestra.run.smithi204.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:31:11.286 INFO:teuthology.orchestra.run.smithi204.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T11:31:11.312 INFO:teuthology.orchestra.run.smithi204.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 631 kB/s | 32 kB 00:00 2024-03-29T11:31:11.337 INFO:teuthology.orchestra.run.smithi204.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2024-03-29T11:31:11.362 INFO:teuthology.orchestra.run.smithi204.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-03-29T11:31:11.388 INFO:teuthology.orchestra.run.smithi204.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:31:11.438 INFO:teuthology.orchestra.run.smithi204.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:31:11.463 INFO:teuthology.orchestra.run.smithi204.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-29T11:31:11.489 INFO:teuthology.orchestra.run.smithi204.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:31:11.514 INFO:teuthology.orchestra.run.smithi204.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:31:11.589 INFO:teuthology.orchestra.run.smithi204.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 125 kB 00:00 2024-03-29T11:31:11.615 INFO:teuthology.orchestra.run.smithi204.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-03-29T11:31:11.790 INFO:teuthology.orchestra.run.smithi204.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 755 kB/s | 151 kB 00:00 2024-03-29T11:31:11.815 INFO:teuthology.orchestra.run.smithi204.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 217 kB/s | 43 kB 00:00 2024-03-29T11:31:11.857 INFO:teuthology.orchestra.run.smithi204.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-29T11:31:11.882 INFO:teuthology.orchestra.run.smithi204.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2024-03-29T11:31:11.908 INFO:teuthology.orchestra.run.smithi204.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-29T11:31:11.933 INFO:teuthology.orchestra.run.smithi204.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-03-29T11:31:11.958 INFO:teuthology.orchestra.run.smithi204.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-03-29T11:31:11.984 INFO:teuthology.orchestra.run.smithi204.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-29T11:31:12.009 INFO:teuthology.orchestra.run.smithi204.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-03-29T11:31:12.034 INFO:teuthology.orchestra.run.smithi204.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-03-29T11:31:12.059 INFO:teuthology.orchestra.run.smithi204.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T11:31:12.085 INFO:teuthology.orchestra.run.smithi204.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-29T11:31:12.102 INFO:teuthology.orchestra.run.smithi204.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 396 kB/s | 16 kB 00:00 2024-03-29T11:31:12.130 INFO:teuthology.orchestra.run.smithi204.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-29T11:31:12.144 INFO:teuthology.orchestra.run.smithi204.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 851 kB/s | 36 kB 00:00 2024-03-29T11:31:12.169 INFO:teuthology.orchestra.run.smithi204.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-03-29T11:31:12.203 INFO:teuthology.orchestra.run.smithi204.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-03-29T11:31:12.228 INFO:teuthology.orchestra.run.smithi204.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 346 kB/s | 20 kB 00:00 2024-03-29T11:31:15.878 INFO:teuthology.orchestra.run.smithi204.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 3.3 MB/s | 17 MB 00:05 2024-03-29T11:31:15.885 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:15.885 INFO:teuthology.orchestra.run.smithi204.stdout:Total 4.4 MB/s | 30 MB 00:06 2024-03-29T11:31:15.910 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:31:15.953 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:31:15.953 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:31:16.371 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:31:16.371 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:31:17.147 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /sys 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /proc 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /mnt 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /var/tmp 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /home 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /root 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /tmp 2024-03-29T11:31:17.148 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:31:17.725 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:31:17.789 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:31:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:31:17.850 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:31:17.921 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:31:17.974 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:31:18.006 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:31:18.082 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:18.098 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:31:18.106 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:18.106 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:18.106 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:31:18.107 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:18.107 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:18.107 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:31:18.108 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:18.177 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:18.177 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:18.185 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:31:18.234 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:31:18.306 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:31:18.360 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:31:18.398 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:31:18.447 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:31:18.503 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:31:18.689 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:31:18.736 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:31:18.767 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:31:18.801 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:31:18.831 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:31:18.908 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:31:18.942 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:31:18.967 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:31:19.007 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:31:19.042 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:31:19.077 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:31:19.166 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:31:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:31:19.280 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:31:19.289 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:19.289 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:31:19.289 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:31:19.290 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:31:19.290 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:31:19.290 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:31:19.290 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:31:19.290 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:19.290 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:31:19.290 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:31:19.291 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:31:19.291 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:31:19.291 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:19.291 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:31:19.291 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:31:19.291 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:31:19.292 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:31:19.292 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:31:19.292 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:31:19.292 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:31:19.292 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:31:19.292 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:31:19.292 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:31:19.293 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:31:19.293 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:31:19.293 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:31:19.293 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:31:19.293 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:31:19.293 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:31:19.293 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:31:19.294 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:31:19.294 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:19.294 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:31:19.294 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:31:19.420 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:31:19.543 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:31:19.707 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:31:19.739 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:31:19.764 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:31:19.764 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:19.764 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:31:19.765 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:31:19.765 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:31:19.765 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:31:20.375 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2024-03-29T11:31:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:31:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:31:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:31:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:31:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:31:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:31:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:31:20.378 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:31:20.667 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-test 2024-03-29T11:31:21.214 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:31:21.238 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:31:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:31:21.241 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:21.241 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:31:21.241 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:31:21.241 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:31:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:21.394 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:31:21.395 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:31:21.395 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:31:21.395 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:31:21.395 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:31:21.395 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:31:21.395 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2024-03-29T11:31:21.396 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:31:21.397 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 48 M 2024-03-29T11:31:21.397 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 204 M 2024-03-29T11:31:21.397 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:31:21.671 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-29T11:31:21.682 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:31:21.706 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:31:21.706 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:21.706 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:31:21.706 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:31:21.706 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:31:21.706 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:21.738 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-29T11:31:21.780 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T11:31:21.925 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:31:21.947 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-29T11:31:21.947 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:31:21.947 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:21.947 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:31:21.948 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:31:21.948 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:31:21.948 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:22.013 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:31:22.047 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 316 kB/s | 163 kB 00:00 2024-03-29T11:31:23.781 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:31:23.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:31:23.783 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:31:23.784 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:31:23.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:31:23.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:31:23.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:31:23.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:31:23.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:31:23.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:31:23.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:31:23.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:31:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:31:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:24.694 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:31:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:24.694 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:31:24.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:31:24.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:31:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:31:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:31:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:31:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:31:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:24.697 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:31:25.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base 2024-03-29T11:31:25.659 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:31:25.757 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:25.811 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:31:25.812 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:31:25.812 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:31:25.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install cephadm 2024-03-29T11:31:26.444 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:31:26.594 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-29T11:31:26.595 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:26.596 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 747 k 2024-03-29T11:31:26.596 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 754 k 2024-03-29T11:31:26.596 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:31:28.053 INFO:teuthology.orchestra.run.smithi204.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 513 kB/s | 747 kB 00:01 2024-03-29T11:31:28.053 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:28.054 INFO:teuthology.orchestra.run.smithi204.stdout:Total 512 kB/s | 747 kB 00:01 2024-03-29T11:31:28.054 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:31:28.060 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:31:28.060 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:31:28.066 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:31:28.066 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:31:28.147 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:31:28.567 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:31:28.673 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:31:29.057 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:31:29.358 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:31:29.359 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:29.359 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:31:29.359 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:29.359 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:29.359 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:31:29.574 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:31:30.149 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:31:30.314 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-29T11:31:30.315 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:30.316 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 145 k 2024-03-29T11:31:30.316 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 436 k 2024-03-29T11:31:30.316 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:31:31.084 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 189 kB/s | 145 kB 00:00 2024-03-29T11:31:31.085 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:31.085 INFO:teuthology.orchestra.run.smithi204.stdout:Total 189 kB/s | 145 kB 00:00 2024-03-29T11:31:31.085 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:31:31.093 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:31:31.093 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:31:31.148 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:31:31.149 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:31:31.284 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:31:31.352 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:31:31.376 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:31:31.377 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:31.377 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:31:31.377 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:32.331 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:31:32.331 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:32.331 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:31:32.331 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:32.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:32.332 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:31:32.576 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr 2024-03-29T11:31:33.153 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:31:33.251 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:33.303 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:31:33.304 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:31:33.304 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:31:33.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:31:33.952 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:31:34.102 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:31:34.103 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-29T11:31:34.103 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:31:34.103 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-29T11:31:34.103 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:31:34.105 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:34.105 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:31:34.105 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-29T11:31:34.105 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-03-29T11:31:34.105 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:34.105 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 6.9 M 2024-03-29T11:31:34.105 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 26 M 2024-03-29T11:31:34.106 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:31:34.939 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 78 kB/s | 31 kB 00:00 2024-03-29T11:31:35.031 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 31 kB/s | 15 kB 00:00 2024-03-29T11:31:35.098 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.3 MB/s | 1.0 MB 00:00 2024-03-29T11:31:35.123 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-29T11:31:35.299 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.1 MB/s | 548 kB 00:00 2024-03-29T11:31:35.324 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.7 MB/s | 863 kB 00:00 2024-03-29T11:31:35.349 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 387 kB/s | 19 kB 00:00 2024-03-29T11:31:35.399 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-29T11:31:35.450 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 615 kB/s | 31 kB 00:00 2024-03-29T11:31:35.500 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-29T11:31:35.567 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.4 MB/s | 2.0 MB 00:00 2024-03-29T11:31:37.509 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 614 kB/s | 1.8 MB 00:02 2024-03-29T11:31:37.512 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:37.512 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.0 MB/s | 6.9 MB 00:03 2024-03-29T11:31:37.533 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:31:37.553 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:31:37.553 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:31:37.742 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:31:37.742 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:31:38.270 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:31:38.351 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:31:38.394 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:31:38.441 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:31:38.479 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:31:38.588 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:31:38.714 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:31:38.808 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:31:38.851 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:31:38.879 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:31:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:31:39.162 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:31:39.187 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:31:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:31:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:31:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:31:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:31:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:31:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:31:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:31:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:31:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:31:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:31:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:31:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:31:40.357 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:31:40.358 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:31:40.358 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:31:40.358 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:31:40.358 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:40.358 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:31:40.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:31:41.186 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:31:41.383 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:31:41.383 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-29T11:31:41.383 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:31:41.383 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-29T11:31:41.383 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:31:41.383 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:41.384 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:31:41.385 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-29T11:31:41.385 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-03-29T11:31:41.385 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:41.385 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 37 M 2024-03-29T11:31:41.385 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 209 M 2024-03-29T11:31:41.385 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:31:41.617 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2024-03-29T11:31:41.675 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 295 kB/s | 17 kB 00:00 2024-03-29T11:31:41.785 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 315 kB/s | 34 kB 00:00 2024-03-29T11:31:42.768 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.3 MB/s | 3.0 MB 00:01 2024-03-29T11:31:42.926 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.1 MB/s | 4.7 MB 00:01 2024-03-29T11:31:43.143 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-29T11:31:43.168 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 905 kB/s | 217 kB 00:00 2024-03-29T11:31:43.268 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 7.9 MB/s | 803 kB 00:00 2024-03-29T11:31:43.335 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 195 kB 00:00 2024-03-29T11:31:43.577 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-29T11:31:43.702 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 673 kB/s | 245 kB 00:00 2024-03-29T11:31:45.511 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.8 MB/s | 7.4 MB 00:04 2024-03-29T11:31:45.514 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:45.514 INFO:teuthology.orchestra.run.smithi204.stdout:Total 8.9 MB/s | 37 MB 00:04 2024-03-29T11:31:45.559 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:31:45.585 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:31:45.585 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:31:45.950 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:31:45.950 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:31:46.664 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:31:46.725 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:31:46.757 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:31:46.852 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:31:46.894 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:31:47.073 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:31:47.101 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:31:47.187 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:31:47.849 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:31:47.965 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:31:49.218 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:31:50.278 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:31:50.309 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:31:52.856 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:31:52.856 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:31:53.260 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:31:53.260 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:53.260 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:31:53.262 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:31:53.262 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:31:53.262 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:31:53.263 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:31:53.263 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:31:53.263 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:31:53.263 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:31:53.263 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:31:53.264 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:31:53.265 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:31:53.265 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:31:53.265 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:31:53.265 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:53.265 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:31:53.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:31:54.224 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:31:54.376 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:31:54.377 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:31:54.378 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout:Install 13 Packages 2024-03-29T11:31:54.379 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:31:54.380 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.0 M 2024-03-29T11:31:54.380 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 26 M 2024-03-29T11:31:54.380 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:31:54.844 INFO:teuthology.orchestra.run.smithi204.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 227 kB/s | 24 kB 00:00 2024-03-29T11:31:54.895 INFO:teuthology.orchestra.run.smithi204.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-29T11:31:54.936 INFO:teuthology.orchestra.run.smithi204.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T11:31:54.962 INFO:teuthology.orchestra.run.smithi204.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:31:54.986 INFO:teuthology.orchestra.run.smithi204.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:31:55.011 INFO:teuthology.orchestra.run.smithi204.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 868 kB/s | 43 kB 00:00 2024-03-29T11:31:55.136 INFO:teuthology.orchestra.run.smithi204.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-03-29T11:31:55.270 INFO:teuthology.orchestra.run.smithi204.stdout:(8/13): python3-google-auth-2.28.2-1.el9.noarch 1.5 MB/s | 203 kB 00:00 2024-03-29T11:31:55.304 INFO:teuthology.orchestra.run.smithi204.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 92 kB/s | 52 kB 00:00 2024-03-29T11:31:55.612 INFO:teuthology.orchestra.run.smithi204.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-29T11:31:55.704 INFO:teuthology.orchestra.run.smithi204.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 978 kB/s | 90 kB 00:00 2024-03-29T11:31:56.172 INFO:teuthology.orchestra.run.smithi204.stdout:(12/13): python3-certifi-2021.10.8-2.el9.noarch 13 kB/s | 15 kB 00:01 2024-03-29T11:31:58.798 INFO:teuthology.orchestra.run.smithi204.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 17 kB/s | 59 kB 00:03 2024-03-29T11:31:58.802 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:58.802 INFO:teuthology.orchestra.run.smithi204.stdout:Total 475 kB/s | 2.0 MB 00:04 2024-03-29T11:31:58.802 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:31:58.816 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:31:58.816 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:31:58.914 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:31:58.918 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:31:59.443 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:31:59.627 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:31:59.689 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:31:59.744 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:31:59.783 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:31:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:31:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:32:00.017 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:32:00.067 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:32:00.784 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:32:00.831 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:32:00.864 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:32:00.913 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:32:00.934 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:32:01.785 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:32:01.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:32:01.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:32:01.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:32:01.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:32:01.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:32:01.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:32:01.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:32:01.786 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:32:01.786 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:32:01.786 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:32:01.786 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:32:01.786 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:32:02.148 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:32:02.148 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:02.148 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:02.148 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:02.148 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:32:02.149 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:32:02.149 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:32:02.149 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:32:02.149 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:32:02.149 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:32:02.149 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:32:02.149 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:32:02.150 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:32:02.150 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:32:02.150 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:32:02.150 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:32:02.150 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:02.150 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:02.396 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:32:02.969 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:03.120 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:32:03.121 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:03.122 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:32:03.122 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-29T11:32:03.122 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-03-29T11:32:03.122 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:03.122 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 6.8 M 2024-03-29T11:32:03.122 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 33 M 2024-03-29T11:32:03.122 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:32:03.866 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.1 MB/s | 248 kB 00:00 2024-03-29T11:32:04.166 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T11:32:04.518 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 180 kB/s | 138 kB 00:00 2024-03-29T11:32:04.650 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 119 kB/s | 58 kB 00:00 2024-03-29T11:32:04.675 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 698 kB/s | 548 kB 00:00 2024-03-29T11:32:05.034 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 99 kB/s | 51 kB 00:00 2024-03-29T11:32:05.037 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:05.038 INFO:teuthology.orchestra.run.smithi204.stdout:Total 3.5 MB/s | 6.8 MB 00:01 2024-03-29T11:32:05.051 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:32:05.062 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:32:05.062 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:32:05.151 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:32:05.153 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:32:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:32:05.634 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:32:05.677 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:32:06.209 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:32:06.306 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:32:06.368 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:32:06.400 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:32:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:32:07.194 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:32:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:32:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:32:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:32:07.195 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:32:07.491 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:32:07.491 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:07.492 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:07.492 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:07.492 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:32:07.492 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:32:07.492 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:32:07.492 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:32:07.493 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:32:07.493 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:07.493 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:07.716 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse 2024-03-29T11:32:08.283 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:08.433 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:08.434 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:08.434 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:32:08.434 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:08.434 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:32:08.434 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:32:08.434 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:32:08.434 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:32:08.434 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:08.435 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:32:08.435 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:08.435 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-03-29T11:32:08.435 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:08.435 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 964 k 2024-03-29T11:32:08.435 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.8 M 2024-03-29T11:32:08.435 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:32:08.571 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 583 kB/s | 79 kB 00:00 2024-03-29T11:32:08.821 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 885 kB 00:00 2024-03-29T11:32:08.824 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:08.824 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.4 MB/s | 964 kB 00:00 2024-03-29T11:32:08.825 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:32:08.845 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:32:08.845 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:32:08.919 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:32:08.919 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:32:09.100 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:32:09.233 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:32:09.304 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:32:09.954 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:32:09.954 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:32:10.245 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:32:10.245 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:10.245 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:10.245 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:32:10.245 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:10.245 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:10.534 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-volume 2024-03-29T11:32:11.118 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:11.280 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:11.281 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:32:11.282 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:11.282 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-03-29T11:32:11.282 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:11.282 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 332 k 2024-03-29T11:32:11.282 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.5 M 2024-03-29T11:32:11.282 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:32:11.411 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 539 kB/s | 69 kB 00:00 2024-03-29T11:32:11.778 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 533 kB/s | 264 kB 00:00 2024-03-29T11:32:11.779 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:11.780 INFO:teuthology.orchestra.run.smithi204.stdout:Total 668 kB/s | 332 kB 00:00 2024-03-29T11:32:11.780 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:32:11.788 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:32:11.788 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:32:11.826 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:32:11.826 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:32:11.983 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:32:12.177 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:32:12.222 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:32:12.238 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:32:12.239 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:12.239 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:32:12.239 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:12.426 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 951 kB/s | 47 MB 00:50 2024-03-29T11:32:12.428 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:12.428 INFO:teuthology.orchestra.run.smithi022.stdout:Total 967 kB/s | 48 MB 00:51 2024-03-29T11:32:12.440 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:32:12.462 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:32:12.462 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:32:12.865 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:32:12.865 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:32:12.986 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:32:13.343 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:13.344 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:13.466 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:32:13.544 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:13.585 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:13.625 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:13.668 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel 2024-03-29T11:32:13.673 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:13.695 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:14.245 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:14.395 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:14.396 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:14.396 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 127 k 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 456 k 2024-03-29T11:32:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:32:15.059 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 191 kB/s | 127 kB 00:00 2024-03-29T11:32:15.060 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:15.060 INFO:teuthology.orchestra.run.smithi204.stdout:Total 191 kB/s | 127 kB 00:00 2024-03-29T11:32:15.060 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:32:15.066 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:32:15.066 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:32:15.124 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:32:15.125 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:32:15.279 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:32:15.375 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:15.831 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:16.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:16.238 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:16.238 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:16.238 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:16.239 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:16.239 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:16.353 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2 2024-03-29T11:32:16.939 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:17.043 INFO:teuthology.orchestra.run.smithi204.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:17.098 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:17.099 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:32:17.099 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:17.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs-devel 2024-03-29T11:32:17.738 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:17.890 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:17.890 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:17.890 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-29T11:32:17.890 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:17.891 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:32:17.891 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:32:17.891 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:17.891 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:32:17.891 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:17.891 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-29T11:32:17.891 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:17.892 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 31 k 2024-03-29T11:32:17.892 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 140 k 2024-03-29T11:32:17.893 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:32:18.321 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 72 kB/s | 31 kB 00:00 2024-03-29T11:32:18.322 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:18.323 INFO:teuthology.orchestra.run.smithi204.stdout:Total 72 kB/s | 31 kB 00:00 2024-03-29T11:32:18.323 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:32:18.326 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:32:18.326 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:32:18.356 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:32:18.357 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:32:18.394 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:18.481 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:18.488 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:32:18.587 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:18.938 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:19.199 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:19.200 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:19.200 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:19.200 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:19.200 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:19.200 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:19.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2 2024-03-29T11:32:19.936 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:20.035 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:20.088 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:20.089 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:32:20.089 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:20.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1 2024-03-29T11:32:20.702 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:20.800 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:20.852 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:20.853 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:32:20.853 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:20.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi022.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi022.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:32:21.473 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:21.554 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph 2024-03-29T11:32:21.570 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:21.624 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:21.625 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:32:21.625 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:21.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw 2024-03-29T11:32:22.122 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:32:22.267 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:22.281 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:22.288 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:22.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:22.290 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:22.291 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:22.291 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:22.291 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:22.291 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:22.291 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:22.291 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:22.291 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:22.292 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:22.292 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:22.292 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:22.292 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:22.292 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:22.293 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:32:22.293 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-29T11:32:22.293 INFO:teuthology.orchestra.run.smithi022.stdout:Install 36 Packages 2024-03-29T11:32:22.293 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:22.293 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 30 M 2024-03-29T11:32:22.293 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 107 M 2024-03-29T11:32:22.293 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:32:22.368 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:22.420 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:22.420 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:32:22.421 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:22.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs 2024-03-29T11:32:22.963 INFO:teuthology.orchestra.run.smithi022.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-29T11:32:23.060 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:23.159 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:23.211 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:23.211 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:32:23.211 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:23.273 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd 2024-03-29T11:32:23.839 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:23.937 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:23.989 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:23.990 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-29T11:32:23.990 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:24.042 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse 2024-03-29T11:32:24.608 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:24.757 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:24.757 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 85 k 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 231 k 2024-03-29T11:32:24.758 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:32:24.972 INFO:teuthology.orchestra.run.smithi022.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 641 kB/s | 1.5 MB 00:02 2024-03-29T11:32:25.263 INFO:teuthology.orchestra.run.smithi022.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 814 kB/s | 2.1 MB 00:02 2024-03-29T11:32:25.341 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 146 kB/s | 85 kB 00:00 2024-03-29T11:32:25.342 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:25.342 INFO:teuthology.orchestra.run.smithi204.stdout:Total 146 kB/s | 85 kB 00:00 2024-03-29T11:32:25.342 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:32:25.350 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:32:25.350 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:32:25.355 INFO:teuthology.orchestra.run.smithi022.stdout:(4/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 4.7 MB 00:02 2024-03-29T11:32:25.406 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:32:25.407 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:32:25.472 INFO:teuthology.orchestra.run.smithi022.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:32:25.514 INFO:teuthology.orchestra.run.smithi022.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:32:25.548 INFO:teuthology.orchestra.run.smithi022.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-03-29T11:32:25.549 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:32:25.581 INFO:teuthology.orchestra.run.smithi022.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:32:25.607 INFO:teuthology.orchestra.run.smithi022.stdout:(9/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 721 kB/s | 247 kB 00:00 2024-03-29T11:32:25.624 INFO:teuthology.orchestra.run.smithi022.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 922 kB/s | 39 kB 00:00 2024-03-29T11:32:25.649 INFO:teuthology.orchestra.run.smithi022.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:32:25.658 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:25.699 INFO:teuthology.orchestra.run.smithi022.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:32:25.725 INFO:teuthology.orchestra.run.smithi022.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-29T11:32:25.750 INFO:teuthology.orchestra.run.smithi022.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:32:25.775 INFO:teuthology.orchestra.run.smithi022.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:32:25.859 INFO:teuthology.orchestra.run.smithi022.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-29T11:32:25.926 INFO:teuthology.orchestra.run.smithi022.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 713 kB/s | 125 kB 00:00 2024-03-29T11:32:26.041 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:26.226 INFO:teuthology.orchestra.run.smithi022.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-03-29T11:32:26.301 INFO:teuthology.orchestra.run.smithi022.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 342 kB/s | 151 kB 00:00 2024-03-29T11:32:26.326 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:26.326 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:26.326 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:26.326 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:26.327 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:26.327 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:26.502 INFO:teuthology.orchestra.run.smithi022.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 623 kB/s | 172 kB 00:00 2024-03-29T11:32:26.535 INFO:teuthology.orchestra.run.smithi022.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-29T11:32:26.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2024-03-29T11:32:26.594 INFO:teuthology.orchestra.run.smithi022.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-29T11:32:26.628 INFO:teuthology.orchestra.run.smithi022.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-29T11:32:26.694 INFO:teuthology.orchestra.run.smithi022.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-29T11:32:26.720 INFO:teuthology.orchestra.run.smithi022.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-29T11:32:26.795 INFO:teuthology.orchestra.run.smithi022.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-29T11:32:26.820 INFO:teuthology.orchestra.run.smithi022.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2024-03-29T11:32:26.912 INFO:teuthology.orchestra.run.smithi022.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 675 kB/s | 79 kB 00:00 2024-03-29T11:32:26.954 INFO:teuthology.orchestra.run.smithi022.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-29T11:32:27.013 INFO:teuthology.orchestra.run.smithi022.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-29T11:32:27.055 INFO:teuthology.orchestra.run.smithi022.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 897 kB/s | 90 kB 00:00 2024-03-29T11:32:27.135 INFO:teuthology.orchestra.run.smithi022.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 294 kB/s | 36 kB 00:00 2024-03-29T11:32:27.169 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:27.183 INFO:teuthology.orchestra.run.smithi022.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-29T11:32:27.272 INFO:teuthology.orchestra.run.smithi022.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-29T11:32:27.323 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:27.324 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:27.324 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-29T11:32:27.324 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:27.324 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:32:27.324 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:32:27.325 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:27.325 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:32:27.325 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:27.325 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-29T11:32:27.325 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:27.325 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.1 M 2024-03-29T11:32:27.325 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 13 M 2024-03-29T11:32:27.325 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:32:27.347 INFO:teuthology.orchestra.run.smithi022.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.0 MB/s | 427 kB 00:00 2024-03-29T11:32:31.162 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 824 kB/s | 3.1 MB 00:03 2024-03-29T11:32:31.162 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:31.163 INFO:teuthology.orchestra.run.smithi204.stdout:Total 824 kB/s | 3.1 MB 00:03 2024-03-29T11:32:31.163 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:32:31.174 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:32:31.175 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:32:31.252 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:32:31.252 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:32:31.684 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:32:31.754 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:31.779 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:31.779 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:31.779 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:32:31.779 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:32:31.780 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:32:31.780 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:32.924 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:32.924 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:32.925 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:32.925 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:32.925 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:32.925 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:33.171 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd 2024-03-29T11:32:33.748 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 11:30:26 AM UTC. 2024-03-29T11:32:33.897 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-29T11:32:33.897 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:33.897 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-29T11:32:33.898 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:33.898 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-29T11:32:33.898 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:32:33.898 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:33.898 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-29T11:32:33.898 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-29T11:32:33.898 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-29T11:32:33.898 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:33.899 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 171 k 2024-03-29T11:32:33.899 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 494 k 2024-03-29T11:32:33.899 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-29T11:32:34.724 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 208 kB/s | 171 kB 00:00 2024-03-29T11:32:34.725 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:34.725 INFO:teuthology.orchestra.run.smithi204.stdout:Total 207 kB/s | 171 kB 00:00 2024-03-29T11:32:34.725 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-29T11:32:34.733 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-29T11:32:34.734 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-29T11:32:34.791 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-29T11:32:34.791 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-29T11:32:34.949 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-29T11:32:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:35.774 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:32:35.774 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:35.774 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-29T11:32:35.775 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:35.775 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:32:35.775 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-29T11:32:36.004 DEBUG:teuthology.parallel:result is None 2024-03-29T11:32:40.913 INFO:teuthology.orchestra.run.smithi022.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 17 MB 00:15 2024-03-29T11:32:40.917 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:40.917 INFO:teuthology.orchestra.run.smithi022.stdout:Total 1.6 MB/s | 30 MB 00:18 2024-03-29T11:32:40.941 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:32:40.987 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:32:40.988 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:32:41.403 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:32:41.404 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:32:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:32:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:32:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:32:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:32:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:32:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:32:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:32:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:32:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:32:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:32:43.386 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:32:43.438 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:32:43.475 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:32:43.534 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:32:43.597 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:32:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:32:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:32:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:32:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:32:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:32:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:32:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:32:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:32:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:32:44.211 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:32:44.305 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:32:44.342 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:32:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:32:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:32:44.700 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:44.866 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:32:44.896 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:44.922 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:44.922 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:44.922 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:32:44.922 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:44.922 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:44.922 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:46.431 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:46.457 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:46.457 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:46.457 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:32:46.457 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:46.457 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:46.457 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:46.915 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:46.938 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:46.938 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:46.938 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:32:46.938 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:46.938 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:46.938 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:47.157 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:47.179 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:47.180 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:47.180 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:32:47.180 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:47.180 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:47.180 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:47.248 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:48.973 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:48.973 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:32:48.974 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:32:48.975 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:32:48.976 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:32:48.976 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:32:48.977 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:32:48.977 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:32:48.977 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:48.977 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:32:48.977 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:32:48.977 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:32:48.977 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:32:48.977 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:32:49.989 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:32:49.989 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:49.989 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:32:49.989 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:49.989 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:49.989 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:32:49.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:32:49.991 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:32:49.992 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:32:49.992 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:32:49.992 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:32:49.992 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:32:49.992 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:49.992 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:32:50.344 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-base 2024-03-29T11:32:50.917 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:32:51.019 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:51.075 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:32:51.076 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:32:51.076 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:32:51.146 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install cephadm 2024-03-29T11:32:51.718 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:32:51.874 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:32:51.875 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:32:51.875 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:32:51.875 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:32:51.875 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 747 k 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 754 k 2024-03-29T11:32:51.876 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:32:52.376 INFO:teuthology.orchestra.run.smithi022.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.5 MB/s | 747 kB 00:00 2024-03-29T11:32:52.376 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:52.376 INFO:teuthology.orchestra.run.smithi022.stdout:Total 1.5 MB/s | 747 kB 00:00 2024-03-29T11:32:52.376 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:32:52.382 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:32:52.383 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:32:52.388 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:32:52.389 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:32:52.488 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:32:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:53.644 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:32:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:53.645 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:32:53.939 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:32:54.510 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:32:54.666 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:32:54.666 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:32:54.666 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 145 k 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 436 k 2024-03-29T11:32:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:32:55.081 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 351 kB/s | 145 kB 00:00 2024-03-29T11:32:55.082 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:55.082 INFO:teuthology.orchestra.run.smithi022.stdout:Total 350 kB/s | 145 kB 00:00 2024-03-29T11:32:55.082 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:32:55.091 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:32:55.091 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:32:55.146 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:32:55.146 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:32:55.294 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:32:55.374 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:55.398 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:55.398 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:55.398 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:32:55.398 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:56.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:56.383 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:56.383 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:32:56.383 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:56.383 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:56.383 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:32:56.607 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr 2024-03-29T11:32:57.184 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:32:57.287 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:57.344 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:32:57.345 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:32:57.345 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:32:57.428 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:32:58.001 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:32:58.158 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:32:58.159 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-29T11:32:58.159 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:32:58.159 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-29T11:32:58.159 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:32:58.159 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:32:58.159 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:32:58.159 INFO:teuthology.orchestra.run.smithi022.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:32:58.159 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout:Install 12 Packages 2024-03-29T11:32:58.160 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:32:58.161 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 6.9 M 2024-03-29T11:32:58.161 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 26 M 2024-03-29T11:32:58.161 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:32:58.794 INFO:teuthology.orchestra.run.smithi022.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 72 kB/s | 31 kB 00:00 2024-03-29T11:32:58.852 INFO:teuthology.orchestra.run.smithi022.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 31 kB/s | 15 kB 00:00 2024-03-29T11:32:58.961 INFO:teuthology.orchestra.run.smithi022.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-29T11:32:58.986 INFO:teuthology.orchestra.run.smithi022.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 238 kB 00:00 2024-03-29T11:32:59.462 INFO:teuthology.orchestra.run.smithi022.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.6 MB/s | 1.8 MB 00:01 2024-03-29T11:32:59.662 INFO:teuthology.orchestra.run.smithi022.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 97 kB/s | 19 kB 00:00 2024-03-29T11:32:59.787 INFO:teuthology.orchestra.run.smithi022.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.0 MB/s | 863 kB 00:00 2024-03-29T11:32:59.812 INFO:teuthology.orchestra.run.smithi022.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 664 kB/s | 548 kB 00:00 2024-03-29T11:32:59.880 INFO:teuthology.orchestra.run.smithi022.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-29T11:33:00.055 INFO:teuthology.orchestra.run.smithi022.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.2 MB/s | 2.0 MB 00:00 2024-03-29T11:33:00.080 INFO:teuthology.orchestra.run.smithi022.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 941 kB/s | 188 kB 00:00 2024-03-29T11:33:00.105 INFO:teuthology.orchestra.run.smithi022.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 455 kB/s | 144 kB 00:00 2024-03-29T11:33:00.108 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:00.108 INFO:teuthology.orchestra.run.smithi022.stdout:Total 3.5 MB/s | 6.9 MB 00:01 2024-03-29T11:33:00.128 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:00.148 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:00.148 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:00.330 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:00.331 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:00.848 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:00.948 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:33:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:33:01.039 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:33:01.068 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:33:01.171 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:33:01.299 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:33:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:33:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:33:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:33:01.770 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:33:01.808 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:33:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:33:03.064 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:33:03.064 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:03.064 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:03.064 INFO:teuthology.orchestra.run.smithi022.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:33:03.064 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:03.065 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:03.343 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:33:03.915 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:04.075 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:33:04.076 INFO:teuthology.orchestra.run.smithi022.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout:Install 12 Packages 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:04.078 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 37 M 2024-03-29T11:33:04.078 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 209 M 2024-03-29T11:33:04.078 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:04.337 INFO:teuthology.orchestra.run.smithi022.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 249 kB/s | 32 kB 00:00 2024-03-29T11:33:04.370 INFO:teuthology.orchestra.run.smithi022.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 518 kB/s | 17 kB 00:00 2024-03-29T11:33:04.404 INFO:teuthology.orchestra.run.smithi022.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:33:04.563 INFO:teuthology.orchestra.run.smithi022.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-29T11:33:04.688 INFO:teuthology.orchestra.run.smithi022.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-29T11:33:04.730 INFO:teuthology.orchestra.run.smithi022.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-29T11:33:04.788 INFO:teuthology.orchestra.run.smithi022.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-29T11:33:05.072 INFO:teuthology.orchestra.run.smithi022.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-29T11:33:05.105 INFO:teuthology.orchestra.run.smithi022.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-29T11:33:05.156 INFO:teuthology.orchestra.run.smithi022.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-03-29T11:33:05.231 INFO:teuthology.orchestra.run.smithi022.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-29T11:33:07.498 INFO:teuthology.orchestra.run.smithi022.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.2 MB/s | 7.4 MB 00:03 2024-03-29T11:33:07.500 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:07.500 INFO:teuthology.orchestra.run.smithi022.stdout:Total 11 MB/s | 37 MB 00:03 2024-03-29T11:33:07.544 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:07.571 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:07.571 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:07.957 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:07.957 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:08.679 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:08.746 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:08.777 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:08.876 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:08.922 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:09.108 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:09.138 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:09.222 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:09.916 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:10.029 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:11.348 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:14.894 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:14.894 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:14.894 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:14.894 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:14.894 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:14.894 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:14.894 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:14.895 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:14.895 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:14.895 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:14.895 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:14.895 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:15.340 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:15.728 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:16.303 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:16.460 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:16.462 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout:Install 13 Packages 2024-03-29T11:33:16.463 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:16.464 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 2.0 M 2024-03-29T11:33:16.464 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 26 M 2024-03-29T11:33:16.464 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:16.637 INFO:teuthology.orchestra.run.smithi022.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 197 kB/s | 24 kB 00:00 2024-03-29T11:33:16.662 INFO:teuthology.orchestra.run.smithi022.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-03-29T11:33:16.713 INFO:teuthology.orchestra.run.smithi022.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:33:16.738 INFO:teuthology.orchestra.run.smithi022.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:33:16.763 INFO:teuthology.orchestra.run.smithi022.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:33:16.780 INFO:teuthology.orchestra.run.smithi022.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:33:16.797 INFO:teuthology.orchestra.run.smithi022.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 184 kB/s | 52 kB 00:00 2024-03-29T11:33:16.856 INFO:teuthology.orchestra.run.smithi022.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-03-29T11:33:16.873 INFO:teuthology.orchestra.run.smithi022.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 295 kB/s | 32 kB 00:00 2024-03-29T11:33:16.940 INFO:teuthology.orchestra.run.smithi022.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-03-29T11:33:16.956 INFO:teuthology.orchestra.run.smithi022.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-03-29T11:33:16.990 INFO:teuthology.orchestra.run.smithi022.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-29T11:33:17.049 INFO:teuthology.orchestra.run.smithi022.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-03-29T11:33:17.050 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:17.051 INFO:teuthology.orchestra.run.smithi022.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2024-03-29T11:33:17.051 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:17.066 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:17.066 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:17.167 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:17.169 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:17.686 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:17.872 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:17.928 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:17.983 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:18.018 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:18.060 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:18.166 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:18.275 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:18.329 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:19.095 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:19.143 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:19.175 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:19.224 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:19.253 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:20.421 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:20.421 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:20.421 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:20.421 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:20.421 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:20.421 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:20.422 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:20.652 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:21.233 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:21.391 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:21.688 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T11:33:21.689 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2024-03-29T11:33:21.690 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:21.690 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 6.8 M 2024-03-29T11:33:21.690 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 33 M 2024-03-29T11:33:21.690 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:22.018 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-03-29T11:33:22.285 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T11:33:22.377 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 629 kB/s | 58 kB 00:00 2024-03-29T11:33:22.402 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-03-29T11:33:22.428 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-29T11:33:24.228 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 59 kB/s | 138 kB 00:02 2024-03-29T11:33:24.229 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:24.230 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.4 MB/s | 6.8 MB 00:02 2024-03-29T11:33:24.245 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:24.257 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:24.257 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:24.353 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:24.354 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:24.654 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:24.880 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:25.431 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:26.702 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:26.703 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:26.703 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:26.926 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-fuse 2024-03-29T11:33:27.500 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:47 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:27.656 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:27.656 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:27.656 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:33:27.656 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:27.656 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:27.656 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:27.656 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:33:27.656 INFO:teuthology.orchestra.run.smithi022.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:27.657 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:27.657 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:27.657 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:27.657 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2024-03-29T11:33:27.657 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:27.658 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 964 k 2024-03-29T11:33:27.658 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 2.8 M 2024-03-29T11:33:27.658 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:27.796 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 569 kB/s | 79 kB 00:00 2024-03-29T11:33:28.088 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 885 kB 00:00 2024-03-29T11:33:28.089 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:28.089 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.2 MB/s | 964 kB 00:00 2024-03-29T11:33:28.089 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:28.111 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:28.111 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:28.188 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:28.188 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:28.376 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:29.243 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:29.243 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:29.525 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:29.525 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:29.526 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:29.526 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:29.526 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:29.526 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:29.820 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-volume 2024-03-29T11:33:30.456 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:50 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:30.615 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:30.616 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:30.616 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T11:33:30.616 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:30.616 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 332 k 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 1.5 M 2024-03-29T11:33:30.617 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:30.743 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 547 kB/s | 69 kB 00:00 2024-03-29T11:33:30.902 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 928 kB/s | 264 kB 00:00 2024-03-29T11:33:30.903 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:30.903 INFO:teuthology.orchestra.run.smithi022.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-29T11:33:30.904 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:30.913 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:30.913 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:30.948 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:30.948 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:31.101 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:31.291 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:31.320 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:31.336 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:31.336 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:31.336 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:31.336 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:32.077 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:32.392 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:32.393 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:32.393 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:32.393 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:32.393 INFO:teuthology.orchestra.run.smithi022.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:32.393 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:32.393 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:32.725 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados-devel 2024-03-29T11:33:33.308 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:53 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:33.464 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:33.465 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:33.465 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-29T11:33:33.465 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:33.465 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:33.465 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:33.465 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:33.466 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:33.466 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:33.466 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T11:33:33.466 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:33.466 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 127 k 2024-03-29T11:33:33.466 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 456 k 2024-03-29T11:33:33.466 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:33.840 INFO:teuthology.orchestra.run.smithi022.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 338 kB/s | 127 kB 00:00 2024-03-29T11:33:33.841 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:33.841 INFO:teuthology.orchestra.run.smithi022.stdout:Total 337 kB/s | 127 kB 00:00 2024-03-29T11:33:33.841 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:33.847 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:33.848 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:33.906 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:33.906 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:34.063 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:34.178 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:34.658 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:34.968 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:34.968 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:35.182 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs2 2024-03-29T11:33:35.760 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:55 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:35.862 INFO:teuthology.orchestra.run.smithi022.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:35.919 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:35.920 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:33:35.920 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:35.994 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:36.572 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:56 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:36.731 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:36.732 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T11:33:36.733 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:36.733 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 31 k 2024-03-29T11:33:36.733 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 140 k 2024-03-29T11:33:36.733 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:37.230 INFO:teuthology.orchestra.run.smithi022.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 63 kB/s | 31 kB 00:00 2024-03-29T11:33:37.230 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:37.231 INFO:teuthology.orchestra.run.smithi022.stdout:Total 62 kB/s | 31 kB 00:00 2024-03-29T11:33:37.231 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:37.235 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:37.235 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:37.261 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:37.262 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:37.396 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:37.506 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:37.888 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:38.148 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:38.149 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:38.149 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:38.149 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:38.149 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:38.149 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:38.298 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados2 2024-03-29T11:33:38.872 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:58 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:38.972 INFO:teuthology.orchestra.run.smithi022.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:39.027 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:39.028 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:33:39.028 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:39.077 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librbd1 2024-03-29T11:33:39.650 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:59 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:39.750 INFO:teuthology.orchestra.run.smithi022.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:39.807 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:39.808 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:33:39.808 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:39.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rados 2024-03-29T11:33:40.427 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:00 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:40.529 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:33:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:40.626 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rgw 2024-03-29T11:33:41.206 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:01 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:41.308 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:41.363 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:41.364 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:33:41.364 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:41.413 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-cephfs 2024-03-29T11:33:41.992 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:01 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:42.094 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:42.149 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:42.150 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:33:42.150 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:42.197 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rbd 2024-03-29T11:33:42.777 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:02 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:42.879 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:42.934 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:42.935 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T11:33:42.935 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:42.980 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-fuse 2024-03-29T11:33:43.551 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:03 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:43.705 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:43.706 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:43.706 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:43.706 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:43.706 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:43.706 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:33:43.707 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:43.707 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:43.707 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:43.707 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T11:33:43.707 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:43.707 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 85 k 2024-03-29T11:33:43.707 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 231 k 2024-03-29T11:33:43.707 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:43.891 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 464 kB/s | 85 kB 00:00 2024-03-29T11:33:43.892 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:43.892 INFO:teuthology.orchestra.run.smithi022.stdout:Total 461 kB/s | 85 kB 00:00 2024-03-29T11:33:43.892 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:43.899 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:43.899 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:43.958 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:43.958 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:44.948 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:44.948 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:44.949 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:44.949 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:44.949 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:44.949 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:45.176 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-mirror 2024-03-29T11:33:45.755 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:05 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:45.912 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:45.912 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:45.912 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-29T11:33:45.912 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:45.912 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:45.912 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:33:45.912 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:45.912 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:45.913 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:45.913 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T11:33:45.913 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:45.913 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 3.1 M 2024-03-29T11:33:45.913 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 13 M 2024-03-29T11:33:45.913 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:46.322 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.6 MB/s | 3.1 MB 00:00 2024-03-29T11:33:46.323 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:46.324 INFO:teuthology.orchestra.run.smithi022.stdout:Total 7.5 MB/s | 3.1 MB 00:00 2024-03-29T11:33:46.324 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:46.333 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:46.333 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:46.410 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:46.410 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:46.866 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:46.920 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:46.944 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:46.944 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:46.944 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:33:46.944 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:46.944 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:46.944 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:48.123 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:48.123 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:48.353 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-nbd 2024-03-29T11:33:48.932 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:08 ago on Fri 29 Mar 2024 11:30:40 AM UTC. 2024-03-29T11:33:49.085 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T11:33:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:49.088 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 171 k 2024-03-29T11:33:49.088 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 494 k 2024-03-29T11:33:49.088 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T11:33:49.315 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 757 kB/s | 171 kB 00:00 2024-03-29T11:33:49.316 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:49.316 INFO:teuthology.orchestra.run.smithi022.stdout:Total 752 kB/s | 171 kB 00:00 2024-03-29T11:33:49.316 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T11:33:49.323 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T11:33:49.323 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T11:33:49.381 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T11:33:49.382 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T11:33:49.537 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T11:33:49.641 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:50.131 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:50.408 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:50.408 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:50.408 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T11:33:50.409 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:50.409 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:50.409 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T11:33:50.618 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:50.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:33:50.771 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:33:50.794 INFO:teuthology.orchestra.run.smithi022.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:50.795 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:50.795 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:33:50.797 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:33:50.938 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:33:50.965 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:50.965 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:33:50.965 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:33:50.967 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:33:50.967 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:50.967 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:33:50.996 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:33:50.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:33:51.039 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:33:51.039 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:51.039 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:33:51.066 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:33:51.136 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:33:51.136 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:33:51.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:33:51.236 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:33:51.237 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:51.237 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:33:51.268 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:33:51.335 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:33:51.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:33:51.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:33:51.434 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:33:51.435 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:51.435 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:33:51.463 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:33:51.532 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:33:51.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:33:51.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:33:51.631 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:33:51.764 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:33:51.765 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /var/log/ceph 2024-03-29T11:33:51.768 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2024-03-29T11:33:51.795 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:33:51.795 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:33:51.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:33:51.865 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:33:51.865 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:33:51.904 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:33:51.943 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:33:51.943 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:33:51.943 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622717', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi022.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQSUy3yjxAe3hzCysAxj6mi6w9pBA8hIMXdJW134/u3hWyuyWtQiYzvUWmAkLB5se5uJTsiT3NHlNnPnP7UCFE=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvWN01mvFk7+LYeo9d5kf4S4xDRmeshlRoq70u6g/fyxryJ0WCJ9fmGzz1gk3ng0Bf1y7+XgNUOixyAHcUjHUU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:33:51.944 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:33:51.972 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:33:51.988 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:33:52.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:33:52.055 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:52.055 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:33:52.107 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:33:52.107 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2024-03-29T11:33:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:33:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.161 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:33:52.162 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.162 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.162 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-29 11:33:47.223723811 +0000 2024-03-29T11:33:52.162 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-29 11:29:27.174325559 +0000 2024-03-29T11:33:52.162 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-29 11:29:27.174325559 +0000 2024-03-29T11:33:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-29 11:29:27.174325559 +0000 2024-03-29T11:33:52.163 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:33:52.226 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-29T11:33:52.227 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-29T11:33:52.227 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.175e-05 s, 5.6 MB/s 2024-03-29T11:33:52.228 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:33:52.283 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2024-03-29T11:33:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:33:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.339 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:33:52.339 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.339 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.339 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-29 11:33:47.224723799 +0000 2024-03-29T11:33:52.339 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-29 11:29:27.503321258 +0000 2024-03-29T11:33:52.339 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-29 11:29:27.503321258 +0000 2024-03-29T11:33:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-29 11:29:27.503321258 +0000 2024-03-29T11:33:52.340 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:33:52.404 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-29T11:33:52.404 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-29T11:33:52.404 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.4249e-05 s, 5.4 MB/s 2024-03-29T11:33:52.405 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:33:52.461 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2024-03-29T11:33:52.516 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:33:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.517 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:33:52.517 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.517 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.517 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-29 11:33:47.224723799 +0000 2024-03-29T11:33:52.517 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-29 11:29:27.836316906 +0000 2024-03-29T11:33:52.517 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-29 11:29:27.836316906 +0000 2024-03-29T11:33:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-29 11:29:27.836316906 +0000 2024-03-29T11:33:52.517 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:33:52.582 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-29T11:33:52.582 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-29T11:33:52.582 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.6086e-05 s, 5.3 MB/s 2024-03-29T11:33:52.583 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:33:52.639 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2024-03-29T11:33:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:33:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.694 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:33:52.694 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.694 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.694 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-29 11:33:47.225723786 +0000 2024-03-29T11:33:52.694 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-29 11:29:28.163312632 +0000 2024-03-29T11:33:52.694 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-29 11:29:28.163312632 +0000 2024-03-29T11:33:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-29 11:29:28.163312632 +0000 2024-03-29T11:33:52.694 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:33:52.758 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-29T11:33:52.759 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-29T11:33:52.759 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 7.5857e-05 s, 6.7 MB/s 2024-03-29T11:33:52.760 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:33:52.816 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:52.816 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:33:52.816 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:33:52.832 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:33:52.832 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-03-29T11:33:52.887 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:33:52.887 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:52.887 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:33:52.888 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:52.888 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:52.888 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-29 11:32:32.029995675 +0000 2024-03-29T11:33:52.888 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-29 11:29:30.966018740 +0000 2024-03-29T11:33:52.888 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-29 11:29:30.966018740 +0000 2024-03-29T11:33:52.888 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-29 11:29:30.966018740 +0000 2024-03-29T11:33:52.888 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:33:52.953 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-29T11:33:52.954 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-29T11:33:52.954 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000266052 s, 1.9 MB/s 2024-03-29T11:33:52.955 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:33:53.011 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-03-29T11:33:53.067 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:33:53.068 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:53.068 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:33:53.068 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:53.068 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:53.068 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-29 11:32:32.029995675 +0000 2024-03-29T11:33:53.068 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-29 11:29:31.301013850 +0000 2024-03-29T11:33:53.068 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-29 11:29:31.301013850 +0000 2024-03-29T11:33:53.068 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-29 11:29:31.301013850 +0000 2024-03-29T11:33:53.068 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:33:53.133 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-29T11:33:53.133 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-29T11:33:53.134 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000219603 s, 2.3 MB/s 2024-03-29T11:33:53.134 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:33:53.190 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-03-29T11:33:53.245 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:33:53.245 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:53.245 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:33:53.245 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:53.245 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:53.246 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-29 11:32:32.030995661 +0000 2024-03-29T11:33:53.246 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-29 11:29:31.692008143 +0000 2024-03-29T11:33:53.246 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-29 11:29:31.692008143 +0000 2024-03-29T11:33:53.246 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-29 11:29:31.692008143 +0000 2024-03-29T11:33:53.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:33:53.311 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-29T11:33:53.311 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-29T11:33:53.311 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000183593 s, 2.8 MB/s 2024-03-29T11:33:53.312 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:33:53.368 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-03-29T11:33:53.422 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:33:53.423 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:33:53.423 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:33:53.423 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:33:53.423 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:33:53.423 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-29 11:32:32.030995661 +0000 2024-03-29T11:33:53.423 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-29 11:29:32.037003107 +0000 2024-03-29T11:33:53.423 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-29 11:29:32.037003107 +0000 2024-03-29T11:33:53.423 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-29 11:29:32.037003107 +0000 2024-03-29T11:33:53.423 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:33:53.488 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-29T11:33:53.488 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-29T11:33:53.488 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000201641 s, 2.5 MB/s 2024-03-29T11:33:53.489 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:33:53.545 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:53.545 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:33:53.545 INFO:tasks.ceph:Generating config... 2024-03-29T11:33:53.546 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:33:53.546 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:33:53.546 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:33:53.546 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:33:53.546 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:33:53.546 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:33:53.546 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:33:53.546 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:33:53.547 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:33:53.547 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:33:53.547 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:33:53.548 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:33:53.548 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:33:53.548 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:33:53.549 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:33:53.550 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:33:53.550 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:33:53.550 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:33:53.550 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:33:53.550 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:33:53.600 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:33:53.603 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:33:53.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:33:53.675 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.22:6789]'), ('mon.c', '[v1:172.21.15.22:6790]'), ('mon.b', '[v1:172.21.15.204:6789]')] 2024-03-29T11:33:53.676 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.204:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:33:53.677 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:53.677 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:33:53.730 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.22:6789]' --addv c '[v1:172.21.15.22:6790]' --addv b '[v1:172.21.15.204:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:53.815 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:53.815 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: generated fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:33:53.815 INFO:teuthology.orchestra.run.smithi022.stdout:setting min_mon_release = quincy 2024-03-29T11:33:53.815 INFO:teuthology.orchestra.run.smithi022.stdout:epoch 0 2024-03-29T11:33:53.815 INFO:teuthology.orchestra.run.smithi022.stdout:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:33:53.815 INFO:teuthology.orchestra.run.smithi022.stdout:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:33:53.815 INFO:teuthology.orchestra.run.smithi022.stdout:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:33:53.816 INFO:teuthology.orchestra.run.smithi022.stdout:min_mon_release 17 (quincy) 2024-03-29T11:33:53.816 INFO:teuthology.orchestra.run.smithi022.stdout:election_strategy: 1 2024-03-29T11:33:53.816 INFO:teuthology.orchestra.run.smithi022.stdout:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:33:53.816 INFO:teuthology.orchestra.run.smithi022.stdout:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:33:53.816 INFO:teuthology.orchestra.run.smithi022.stdout:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:33:53.816 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:33:53.817 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:33:53.831 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5b52d532-7394-417c-b834-972932886c7f... 2024-03-29T11:33:53.832 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:33:53.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:33:53.921 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: chdir = "" 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: auth supported = cephx 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: filestore xattr use omap = true 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: mon clock drift allowed = 1.000 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: osd crush chooseleaf type = 0 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: auth debug = true 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on old message = true 2024-03-29T11:33:53.922 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on bug = true 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: mon pg warn max object skew = 0 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default size = 2 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow primary affinity = true 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow pg remap = true 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on no sortbitwise = false 2024-03-29T11:33:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on too few osds = false 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: mon_allow_pool_size_one = true 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: osd default data pool replay window = 5 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: mon allow pool delete = true 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: mon cluster log file level = debug 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: debug asserts on shutdown = true 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: mon health detail to clog = false 2024-03-29T11:33:53.924 INFO:teuthology.orchestra.run.smithi022.stdout: mon host = "[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.204:6789]" 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: lockdep = True 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: mon client directed command retry = 5 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: mon election default strategy = 1 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: ms bind msgr2 = False 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject socket failures = 5000 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: ms type = async 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout:[osd] 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: osd journal size = 100 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub load threshold = 5.0 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub max interval = 600 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:33:53.925 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd recover clone overlap = true 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd recovery max chunk = 1048576 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug shutdown = true 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug op order = true 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify stray on activate = true 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug trim objects = true 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd open classes on start = true 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug pg log writeout = true 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.926 INFO:teuthology.orchestra.run.smithi022.stdout: osd map max advance = 10 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: journal zero on create = true 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: filestore apply finisher threads = 3 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: bdev debug aio = true 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug misdirected ops = true 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore block size = 96636764160 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore compression algorithm = snappy 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore compression mode = aggressive 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore fsck on mount = True 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore zero block detection = True 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluefs = 20 2024-03-29T11:33:53.927 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluestore = 20 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: debug monc = 20 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: debug osd = 20 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: debug rocksdb = 10 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd full ratio = 0.9 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: osd class default list = * 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: osd class load list = * 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: osd client watch timeout = 120 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify cached snaps = True 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify missing on start = True 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:33:53.928 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock override recovery settings = True 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: osd objectstore = bluestore 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: osd op queue = debug_random 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: osd op queue cut off = debug_random 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: osd shutdown pgref assert = True 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr] 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: debug mgr = 20 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: mgr/telemetry/nag = false 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.929 INFO:teuthology.orchestra.run.smithi022.stdout:[mon] 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: debug paxos = 20 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: mon data avail warn = 5 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: mon mgr mkfs grace = 240 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd reporter subtree level = osd 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd prime pg temp = true 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.930 INFO:teuthology.orchestra.run.smithi022.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: # 1m isn't quite enough 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:33:53.931 INFO:teuthology.orchestra.run.smithi022.stdout: mon min osdmap epochs = 50 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune interval = 2 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune min = 15 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: mon scrub interval = 300 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on pool no app = False 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: paxos service trim min = 10 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout:[client] 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: rgw cache enabled = true 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable ops log = true 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable usage log = true 2024-03-29T11:33:53.932 INFO:teuthology.orchestra.run.smithi022.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi022.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi022.stdout: debug monc = 20 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi022.stdout: debug objecter = 20 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi022.stdout: debug rados = 20 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.a] 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.c] 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.b] 2024-03-29T11:33:53.933 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-03-29T11:33:53.934 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-03-29T11:33:53.935 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.936 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = "[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.204:6789]" 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: lockdep = True 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: mon election default strategy = 1 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: ms bind msgr2 = False 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2024-03-29T11:33:53.937 INFO:teuthology.orchestra.run.smithi204.stdout: ms type = async 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-03-29T11:33:53.938 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug trim objects = true 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.939 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore compression algorithm = snappy 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore compression mode = aggressive 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2024-03-29T11:33:53.940 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore zero block detection = True 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 20 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 20 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: debug monc = 20 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 10 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: osd class default list = * 2024-03-29T11:33:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: osd class load list = * 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd client watch timeout = 120 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify cached snaps = True 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify missing on start = True 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock override recovery settings = True 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd op queue = debug_random 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd op queue cut off = debug_random 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: osd shutdown pgref assert = True 2024-03-29T11:33:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-03-29T11:33:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:33:53.944 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon min osdmap epochs = 50 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune interval = 2 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune min = 15 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon scrub interval = 300 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on pool no app = False 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: paxos service trim min = 10 2024-03-29T11:33:53.945 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: debug monc = 20 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: debug objecter = 20 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout: debug rados = 20 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.c] 2024-03-29T11:33:53.946 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.b] 2024-03-29T11:33:53.948 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:33:53.948 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:33:54.003 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:33:54.003 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:54.003 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:33:54.017 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:54.017 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:33:54.071 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:33:54.072 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:54.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:33:54.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:33:54.151 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:33:54.151 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:54.206 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:33:54.206 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:33:54.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:33:54.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:33:54.246 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:33:54.246 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:54.300 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:33:54.301 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:33:54.301 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:33:54.355 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:33:54.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:33:54.419 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:33:54.422 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:33:54.422 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:33:54.422 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:33:54.464 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:33:54.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:33:54.522 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:33:54.537 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:33:54.537 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:33:54.537 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:33:54.563 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:54.564 INFO:tasks.ceph:role: osd.0 2024-03-29T11:33:54.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:33:54.564 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:33:54.629 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:33:54.630 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:33:54.662 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-29T11:33:54.664 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-29T11:33:54.664 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:33:54.734 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:33:54.806 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:33:54.873 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:54.873 INFO:tasks.ceph:role: osd.1 2024-03-29T11:33:54.874 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:33:54.874 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:33:54.939 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:33:54.940 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:33:54.973 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-29T11:33:54.974 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-29T11:33:54.974 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:33:55.045 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:33:55.116 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:33:55.183 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:55.183 INFO:tasks.ceph:role: osd.2 2024-03-29T11:33:55.183 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:33:55.183 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:33:55.249 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:33:55.250 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:33:55.250 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:33:55.250 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:33:55.250 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:33:55.250 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:33:55.250 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:33:55.250 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:33:55.251 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:33:55.251 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:33:55.282 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-29T11:33:55.284 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-29T11:33:55.284 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:33:55.354 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:33:55.425 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:33:55.491 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:33:55.492 INFO:tasks.ceph:role: osd.3 2024-03-29T11:33:55.492 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:33:55.492 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:33:55.558 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:33:55.559 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:33:55.591 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-29T11:33:55.592 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-29T11:33:55.592 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:33:55.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:33:55.733 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:55.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:55.836+0000 7fe368019640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:33:55.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:55.836+0000 7fe368019640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:33:55.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:55.837+0000 7fe368019640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:33:55.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:55.837+0000 7fe368019640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:33:55.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:55.837+0000 7fe368019640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:33:55.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:55.837+0000 7fe368019640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:33:58.689 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:33:58.720 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:33:58.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:58.814+0000 7fc8a9b19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:33:58.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:58.814+0000 7fc8a9b19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:33:58.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:58.814+0000 7fc8a9b19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:33:58.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:58.814+0000 7fc8a9b19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:33:58.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:58.814+0000 7fc8a9b19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:33:58.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:33:58.815+0000 7fc8a9b19640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:34:01.661 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:01.692 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:01.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:01.787+0000 7f4be260e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:34:01.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:01.787+0000 7f4be260e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:34:01.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:01.787+0000 7f4be260e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:01.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:01.787+0000 7f4be260e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:01.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:01.787+0000 7f4be260e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:01.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:01.787+0000 7f4be260e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:34:04.627 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:04.658 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:04.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:04.753+0000 7fd6c9e0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:34:04.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:04.753+0000 7fd6c9e0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:34:04.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:04.753+0000 7fd6c9e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:04.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:04.753+0000 7fd6c9e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:04.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:04.753+0000 7fd6c9e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:04.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:04.754+0000 7fd6c9e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:34:07.600 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:07.630 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:07.663 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:07.663 INFO:tasks.ceph:role: osd.4 2024-03-29T11:34:07.663 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:34:07.663 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:07.741 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:07.742 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:08.249 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-29T11:34:08.251 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-29T11:34:08.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:08.367 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:08.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:08.473 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:08.473 INFO:tasks.ceph:role: osd.5 2024-03-29T11:34:08.473 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:34:08.473 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:08.548 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:08.549 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:08.999 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-29T11:34:09.001 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-29T11:34:09.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:09.095 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:09.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:09.193 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:09.193 INFO:tasks.ceph:role: osd.6 2024-03-29T11:34:09.193 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:34:09.193 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:09.268 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:09.268 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:09.268 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:09.268 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:09.269 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:09.269 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:09.269 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:09.755 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-29T11:34:09.757 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-29T11:34:09.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:09.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:09.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:10.012 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:10.012 INFO:tasks.ceph:role: osd.7 2024-03-29T11:34:10.012 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:34:10.012 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:10.087 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:10.087 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:10.087 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:10.087 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:10.087 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:10.087 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:10.087 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:10.088 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:10.088 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:10.088 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:10.593 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-29T11:34:10.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-29T11:34:10.596 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:10.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:10.762 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:10.876 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:10.879+0000 7fe4eb738640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:34:10.876 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:10.879+0000 7fe4eb738640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:34:10.876 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:10.879+0000 7fe4eb738640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:10.876 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:10.879+0000 7fe4eb738640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:10.876 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:10.879+0000 7fe4eb738640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:10.877 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:10.880+0000 7fe4eb738640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:34:13.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:13.753 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:13.846 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:13.848+0000 7fb993cce640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:34:13.846 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:13.849+0000 7fb993cce640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:34:13.846 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:13.849+0000 7fb993cce640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:13.846 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:13.849+0000 7fb993cce640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:13.846 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:13.849+0000 7fb993cce640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:13.846 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:13.849+0000 7fb993cce640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:34:16.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:16.723 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:16.816 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:16.818+0000 7f3fefa11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:34:16.817 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:16.818+0000 7f3fefa11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:34:16.817 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:16.818+0000 7f3fefa11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:16.817 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:16.818+0000 7f3fefa11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:16.817 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:16.818+0000 7f3fefa11640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:16.817 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:16.818+0000 7f3fefa11640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:34:19.666 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:19.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:19.796+0000 7f8e541df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:19.797+0000 7f8e541df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:19.797+0000 7f8e541df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:19.797+0000 7f8e541df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:19.797+0000 7f8e541df640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:19.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:19.797+0000 7f8e541df640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:34:22.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:22.683 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:34:22.683 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:22.683 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:34:22.715 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:22.715 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:34:22.786 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:22.786 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:34:22.857 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:22.857 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:34:22.927 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:22.927 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:34:22.996 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:22.997 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:34:23.029 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:23.029 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:34:23.100 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:23.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:34:23.169 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:23.170 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:34:23.239 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:23.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:34:23.308 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:23.308 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:34:23.326 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:23.326 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:34:23.364 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:34:23.364 DEBUG:teuthology.orchestra.run.smithi022:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:23.368 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:23.407 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.y] 2024-03-29T11:34:23.407 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAipwZmQ2ZuFRAAJxLzQzI76c2UZyN4qMr1Tw== 2024-03-29T11:34:23.407 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.0] 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAjpwZmZBL0MRAANO6RexixT4gQlP0Cvsk15Q== 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.1] 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAmpwZmKxGeMBAA9XIZk94tCTU0z1LLvi9o8Q== 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.2] 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAppwZmdtQBLxAAMfogSnz9Y2ulczojd0xX0g== 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.3] 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAspwZmo/3+LBAArXU52y/fzfNIaSX5vO1sUQ== 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.x] 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAipwZm9T9FGRAAxPGrkD4/DfhTTw+lHOD7Xg== 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.4] 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAypwZmU0J2NBAAsZN8gTcuXAhgBXG6IQ4Spw== 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.5] 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQA1pwZma7mlMhAAaG6C5jVnwhr72FZcS7SC0A== 2024-03-29T11:34:23.408 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.6] 2024-03-29T11:34:23.409 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQA4pwZmanHSMBAAzWHDGY6LWFF7OypDsS4igw== 2024-03-29T11:34:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.7] 2024-03-29T11:34:23.409 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQA7pwZm832QLxAA70ZcXwrsAaEe5C39B7by0g== 2024-03-29T11:34:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2024-03-29T11:34:23.409 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAipwZmV833GxAAqRbQiKacAkKCwMz0nv2Ojw== 2024-03-29T11:34:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:[client.1] 2024-03-29T11:34:23.409 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAipwZmN/NhHxAAXeDQmUez67CzWWSsd+cdnQ== 2024-03-29T11:34:23.430 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.y] 2024-03-29T11:34:23.430 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAipwZmQ2ZuFRAAJxLzQzI76c2UZyN4qMr1Tw== 2024-03-29T11:34:23.430 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2024-03-29T11:34:23.430 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAjpwZmZBL0MRAANO6RexixT4gQlP0Cvsk15Q== 2024-03-29T11:34:23.430 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2024-03-29T11:34:23.430 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAmpwZmKxGeMBAA9XIZk94tCTU0z1LLvi9o8Q== 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAppwZmdtQBLxAAMfogSnz9Y2ulczojd0xX0g== 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.3] 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAspwZmo/3+LBAArXU52y/fzfNIaSX5vO1sUQ== 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAipwZm9T9FGRAAxPGrkD4/DfhTTw+lHOD7Xg== 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.4] 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAypwZmU0J2NBAAsZN8gTcuXAhgBXG6IQ4Spw== 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.5] 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA1pwZma7mlMhAAaG6C5jVnwhr72FZcS7SC0A== 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.6] 2024-03-29T11:34:23.431 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA4pwZmanHSMBAAzWHDGY6LWFF7OypDsS4igw== 2024-03-29T11:34:23.432 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.7] 2024-03-29T11:34:23.432 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQA7pwZm832QLxAA70ZcXwrsAaEe5C39B7by0g== 2024-03-29T11:34:23.432 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2024-03-29T11:34:23.432 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAipwZmV833GxAAqRbQiKacAkKCwMz0nv2Ojw== 2024-03-29T11:34:23.432 INFO:teuthology.orchestra.run.smithi204.stdout:[client.1] 2024-03-29T11:34:23.432 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQAipwZmN/NhHxAAXeDQmUez67CzWWSsd+cdnQ== 2024-03-29T11:34:23.433 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:23.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:23.526 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.528 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.580 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.634 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.636 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.688 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.690 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.742 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:23.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:23.797 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.905 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.958 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:23.961 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:24.013 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:24.015 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:24.075 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:24.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:24.132 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:34:24.132 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:24.159 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:24.255 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:24.281 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:24.349 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:24.445 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:24.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:24.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:24.598 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:24.625 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:24.627 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:24.680 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:34:24.680 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:34:24.680 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:34:24.682 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:34:24.683 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:34:24.683 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:34:24.684 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:34:24.685 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:34:24.685 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:34:24.722 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:34:24.723 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:34:24.723 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:34:24.723 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:34:24.725 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:34:24.725 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:34:24.725 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:34:24.727 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:34:24.727 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:34:24.728 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient: get_monmap_and_config 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient: build_initial_monmap 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient: monmap: 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:34:24.833927+0000 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:34:24.833927+0000 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:24.830 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:24.831 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:24.831 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:24.831 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:24.831 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 1 Processor -- start 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 1 -- start start 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient: init 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient(hunting): picked mon.noname-c con 0x7f680809ac40 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient(hunting): picked mon.noname-a con 0x7f680809c870 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.833+0000 7f680e4d9640 10 monclient(hunting): picked mon.noname-b con 0x7f68080b4360 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.834+0000 7f680e4d9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680809dec0 con 0x7f680809c870 2024-03-29T11:34:24.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.834+0000 7f680e4d9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680809da30 con 0x7f680809ac40 2024-03-29T11:34:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.834+0000 7f680e4d9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68080b39c0 con 0x7f68080b4360 2024-03-29T11:34:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.834+0000 7f680e4d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.834+0000 7f680e4d9640 10 monclient(hunting): authenticate will time out at 915.111037s 2024-03-29T11:34:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.834+0000 7f68077fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f680809c870 0x7f680813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58126/0 (socket says 172.21.15.22:58126) 2024-03-29T11:34:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.834+0000 7f680ca76640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f68080b4360 0x7f680814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34536/0 (socket says 172.21.15.22:34536) 2024-03-29T11:34:24.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:24.834+0000 7f68077fe640 1 -- 172.21.15.22:0/2775197658 learned_addr learned my addr 172.21.15.22:0/2775197658 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 10 monclient(hunting): tick 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 1 monclient(hunting): continuing hunt 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 1 -- 172.21.15.22:0/2775197658 >> v1:172.21.15.22:6790/0 conn(0x7f68080b4360 legacy=0x7f680814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 1 -- 172.21.15.22:0/2775197658 >> v1:172.21.15.204:6789/0 conn(0x7f680809ac40 legacy=0x7f68080b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 1 -- 172.21.15.22:0/2775197658 >> v1:172.21.15.22:6789/0 conn(0x7f680809c870 legacy=0x7f680813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:27.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f67f40036e0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:27.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f67f40072d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:27.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f67f400b020 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:27.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 1 -- 172.21.15.22:0/2775197658 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68080b5160 con 0x7f67f400b020 2024-03-29T11:34:27.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 1 -- 172.21.15.22:0/2775197658 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f400fcb0 con 0x7f67f40036e0 2024-03-29T11:34:27.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 1 -- 172.21.15.22:0/2775197658 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f4010f10 con 0x7f67f40072d0 2024-03-29T11:34:27.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:27.835+0000 7f6805ffb640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:29.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.000+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623068324 0 0) 0x7f68080b5160 con 0x7f67f400b020 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.000+0000 7f6806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.000+0000 7f6806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.000+0000 7f6806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.000+0000 7f6806ffd640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.000+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f0003520 con 0x7f67f400b020 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277892776 0 0) 0x7f67f400fcb0 con 0x7f67f40036e0 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:34:29.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68080b5160 con 0x7f67f40036e0 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910597432 0 0) 0x7f67f4010f10 con 0x7f67f40072d0 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f400fcb0 con 0x7f67f40072d0 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6807fff640 1 -- 172.21.15.22:0/2775197658 >> v1:172.21.15.204:6789/0 conn(0x7f67f40036e0 legacy=0x7f67f4005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6807fff640 1 -- 172.21.15.22:0/2775197658 >> v1:172.21.15.204:6789/0 conn(0x7f67f40036e0 legacy=0x7f67f4005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6807fff640 1 --1- 172.21.15.22:0/2775197658 >> v1:172.21.15.204:6789/0 conn(0x7f67f40036e0 0x7f67f4005b30 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6807fff640 1 --1- 172.21.15.22:0/2775197658 >> v1:172.21.15.204:6789/0 conn(0x7f67f40036e0 0x7f67f4005b30 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:34:29.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.204:6789/0 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3427400406 0 0) 0x7f67f400fcb0 con 0x7f67f40072d0 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 >> v1:172.21.15.204:6789/0 conn(0x7f67f40036e0 legacy=0x7f67f4005b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 >> v1:172.21.15.22:6789/0 conn(0x7f67f400b020 legacy=0x7f67f400d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 monclient: found mon.noname-b 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f4012170 con 0x7f67f40072d0 2024-03-29T11:34:30.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f67fc001450 con 0x7f67f40072d0 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f6806ffd640 10 monclient: dump: 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.007 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f680e4d9640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.001+0000 7f680e4d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.002+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67fc0015b0 con 0x7f67f40072d0 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.002+0000 7f6806ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.002+0000 7f6806ffd640 1 -- 172.21.15.22:0/2775197658 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f67fc001a00 con 0x7f67f40072d0 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.002+0000 7f6806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.002+0000 7f6806ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.002+0000 7f6806ffd640 10 monclient: dump: 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.008 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.002+0000 7f680e4d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.002+0000 7f680e4d9640 4 set_mon_vals no callback set 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.003+0000 7f680e4d9640 10 monclient: shutdown 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.003+0000 7f680e4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.003+0000 7f680e4d9640 1 -- 172.21.15.22:0/2775197658 >> v1:172.21.15.22:6790/0 conn(0x7f67f40072d0 legacy=0x7f67f4009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.003+0000 7f680e4d9640 1 -- 172.21.15.22:0/2775197658 shutdown_connections 2024-03-29T11:34:30.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.003+0000 7f680e4d9640 1 -- 172.21.15.22:0/2775197658 >> 172.21.15.22:0/2775197658 conn(0x7f6808066b00 msgr2=0x7f6808066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.003+0000 7f680e4d9640 1 -- 172.21.15.22:0/2775197658 shutdown_connections 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.003+0000 7f680e4d9640 1 -- 172.21.15.22:0/2775197658 wait complete. 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.004+0000 7f680e4d9640 10 monclient: build_initial_monmap 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.004+0000 7f680e4d9640 10 monclient: monmap: 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:30.010 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.004+0000 7f680e4d9640 1 librados: starting msgr at 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.004+0000 7f680e4d9640 1 librados: starting objecter 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.004+0000 7f680e4d9640 1 Processor -- start 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.005+0000 7f680e4d9640 1 -- start start 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.005+0000 7f680e4d9640 1 librados: setting wanted keys 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.005+0000 7f680e4d9640 1 librados: calling monclient init 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.005+0000 7f680e4d9640 10 monclient: init 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 10 monclient(hunting): picked mon.noname-a con 0x7f67f40036e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6807fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f67f40036e0 0x7f68081da410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58196/0 (socket says 172.21.15.22:58196) 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6807fff640 1 -- 172.21.15.22:0/2135371205 learned_addr learned my addr 172.21.15.22:0/2135371205 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:30.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 10 monclient(hunting): picked mon.noname-c con 0x7f67f40072d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 client.?.objecter ms_handle_connect 0x7f67f40036e0 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 10 monclient(hunting): picked mon.noname-b con 0x7f67f400b020 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68081daf00 con 0x7f67f40036e0 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 client.?.objecter ms_handle_connect 0x7f67f40072d0 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68080b1ac0 con 0x7f67f400b020 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68081dab20 con 0x7f67f40072d0 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f680e4d9640 10 monclient(hunting): authenticate will time out at 920.282779s 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684604845 0 0) 0x7f68081daf00 con 0x7f67f40036e0 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f400a730 con 0x7f67f40036e0 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 client.?.objecter ms_handle_connect 0x7f67f400b020 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.006+0000 7f6804ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3677365671 0 0) 0x7f67f400a730 con 0x7f67f40036e0 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68081daf00 con 0x7f67f40036e0 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f67f80019b0 con 0x7f67f40036e0 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.013 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2540371043 0 0) 0x7f68081daf00 con 0x7f67f40036e0 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 >> v1:172.21.15.22:6790/0 conn(0x7f67f40072d0 legacy=0x7f68080afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 >> v1:172.21.15.204:6789/0 conn(0x7f67f400b020 legacy=0x7f68080b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 monclient: found mon.a 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:30.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6808145a50 con 0x7f67f40036e0 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 10 monclient: _renew_subs 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6808145ca0 con 0x7f67f40036e0 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 10 monclient: _renew_subs 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68081e2940 con 0x7f67f40036e0 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f680e4d9640 1 librados: init done 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f8001c70 con 0x7f67f40036e0 2024-03-29T11:34:30.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.007+0000 7f6804ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.008+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f67f8003a40 con 0x7f67f40036e0 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.008+0000 7f68067fc640 4 set_mon_vals no callback set 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.008+0000 7f6804ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.008+0000 7f6804ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.008+0000 7f6804ff9640 10 monclient: dump: 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.016 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.017 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.017 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:30.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.009+0000 7f680e4d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.009+0000 7f680e4d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.009+0000 7f680e4d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:30.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.009+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e8005080 con 0x7f67f40036e0 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.010+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f67f8003f30 con 0x7f67f40036e0 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.010+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 581478911 0 0) 0x7f67f80065c0 con 0x7f67f40036e0 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.010+0000 7f6804ff9640 10 client.4110.objecter ms_dispatch 0x7f6808000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.010+0000 7f6804ff9640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.010+0000 7f6804ff9640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.010+0000 7f6804ff9640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.012+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f67f80041e0 con 0x7f67f40036e0 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.012+0000 7f6804ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.012+0000 7f6804ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:30.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.087+0000 7f680e4d9640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:30.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.087+0000 7f680e4d9640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:30.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.087+0000 7f680e4d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:30.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.087+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f67e8005330 con 0x7f67f40036e0 2024-03-29T11:34:30.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.089+0000 7f6804ff9640 1 -- 172.21.15.22:0/2135371205 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f67f80057c0 con 0x7f67f40036e0 2024-03-29T11:34:30.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.089+0000 7f6804ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:34:30.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.089+0000 7f6804ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:34:30.085 INFO:teuthology.orchestra.run.smithi022.stderr:adjusted tunables profile to default 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.089+0000 7f680e4d9640 10 librados: watch_flush enter 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 10 librados: watch_flush exit 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 10 monclient: shutdown 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 >> v1:172.21.15.22:6789/0 conn(0x7f67f40036e0 legacy=0x7f68081da410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 shutdown_connections 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 >> 172.21.15.22:0/2135371205 conn(0x7f6808066b00 msgr2=0x7f68080b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 shutdown_connections 2024-03-29T11:34:30.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 1 -- 172.21.15.22:0/2135371205 wait complete. 2024-03-29T11:34:30.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:30.090+0000 7f680e4d9640 1 librados: shutdown 2024-03-29T11:34:30.098 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:34:30.098 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:34:30.098 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:30.098 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:34:30.126 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:30.126 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:34:30.193 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:30.193 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:34:30.260 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T11:34:30.260 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:34:30.327 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:30.327 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:34:30.355 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:30.355 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:34:30.422 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:30.422 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:34:30.486 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:30.489+0000 7f3548438040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:30.489 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-29T11:34:30.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:34:30.494 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:30.498+0000 7f062ea3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:34:30.543 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:30.546+0000 7f3548438040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:30.550 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:30.554+0000 7f062ea3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:34:30.556 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 8a925ac5-9ea5-4bd1-84cf-9415000e9422 0 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.699+0000 7facfaa86640 10 monclient: get_monmap_and_config 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.699+0000 7facfaa86640 10 monclient: build_initial_monmap 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.699+0000 7facfaa86640 10 monclient: monmap: 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:34:30.700082+0000 2024-03-29T11:34:30.696 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:34:30.700082+0000 2024-03-29T11:34:30.697 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:30.697 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.697 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:30.697 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:30.697 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:30.697 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.699+0000 7facfaa86640 1 Processor -- start 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.699+0000 7facfaa86640 1 -- start start 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.699+0000 7facfaa86640 10 monclient: init 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 10 monclient(hunting): picked mon.noname-c con 0x7facf409ac40 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 10 monclient(hunting): picked mon.noname-a con 0x7facf409c870 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 10 monclient(hunting): picked mon.noname-b con 0x7facf40b4360 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf409dec0 con 0x7facf409c870 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf409da30 con 0x7facf409ac40 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf40b39c0 con 0x7facf40b4360 2024-03-29T11:34:30.706 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facfaa86640 10 monclient(hunting): authenticate will time out at 906.966303s 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf8822640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7facf409ac40 0x7facf40b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49892/0 (socket says 172.21.15.204:49892) 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf8822640 1 -- 172.21.15.204:0/825689497 learned_addr learned my addr 172.21.15.204:0/825689497 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479886784 0 0) 0x7facf409da30 con 0x7facf409ac40 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face4003520 con 0x7facf409ac40 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755716903 0 0) 0x7face0001110 con 0x7facf409c870 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:34:30.707 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf409da30 con 0x7facf409c870 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414422831 0 0) 0x7facf40b39c0 con 0x7facf40b4360 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.700+0000 7facf37fe640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face0001110 con 0x7facf40b4360 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1814149045 0 0) 0x7face4003520 con 0x7facf409ac40 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 >> v1:172.21.15.22:6790/0 conn(0x7facf40b4360 legacy=0x7facf414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 >> v1:172.21.15.22:6789/0 conn(0x7facf409c870 legacy=0x7facf413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40b5160 con 0x7facf409ac40 2024-03-29T11:34:30.708 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7face8001690 con 0x7facf409ac40 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: dump: 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.709 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7face8001870 con 0x7facf409ac40 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 1 -- 172.21.15.204:0/825689497 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7face8001c80 con 0x7facf409ac40 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 4 set_mon_vals no callback set 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facf37fe640 10 monclient: dump: 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.710 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 10 monclient: shutdown 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 1 -- 172.21.15.204:0/825689497 >> v1:172.21.15.204:6789/0 conn(0x7facf409ac40 legacy=0x7facf40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 1 -- 172.21.15.204:0/825689497 shutdown_connections 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 1 -- 172.21.15.204:0/825689497 >> 172.21.15.204:0/825689497 conn(0x7facf4066b00 msgr2=0x7facf4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 1 -- 172.21.15.204:0/825689497 shutdown_connections 2024-03-29T11:34:30.711 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.701+0000 7facfaa86640 1 -- 172.21.15.204:0/825689497 wait complete. 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.702+0000 7facfaa86640 10 monclient: build_initial_monmap 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.702+0000 7facfaa86640 10 monclient: monmap: 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.703+0000 7facfaa86640 1 librados: starting msgr at 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.703+0000 7facfaa86640 1 librados: starting objecter 2024-03-29T11:34:30.712 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.703+0000 7facfaa86640 1 Processor -- start 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.703+0000 7facfaa86640 1 -- start start 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.703+0000 7facfaa86640 1 librados: setting wanted keys 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.703+0000 7facfaa86640 1 librados: calling monclient init 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.703+0000 7facfaa86640 10 monclient: init 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 10 monclient(hunting): picked mon.noname-c con 0x7facf409ac40 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 10 monclient(hunting): picked mon.noname-a con 0x7facf40b1280 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 10 monclient(hunting): picked mon.noname-b con 0x7facf41d6ed0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf41422a0 con 0x7facf40b1280 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf4142f50 con 0x7facf41d6ed0 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf41ccab0 con 0x7facf409ac40 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facfaa86640 10 monclient(hunting): authenticate will time out at 906.970349s 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf9023640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7facf41d6ed0 0x7facf413adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49902/0 (socket says 172.21.15.204:49902) 2024-03-29T11:34:30.713 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf9023640 1 -- 172.21.15.204:0/219218093 learned_addr learned my addr 172.21.15.204:0/219218093 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 client.?.objecter ms_handle_connect 0x7facf41d6ed0 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 client.?.objecter ms_handle_connect 0x7facf40b1280 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 client.?.objecter ms_handle_connect 0x7facf409ac40 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366062479 0 0) 0x7facf4142f50 con 0x7facf41d6ed0 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd8004eb0 con 0x7facf41d6ed0 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256714937 0 0) 0x7facf41422a0 con 0x7facf40b1280 2024-03-29T11:34:30.714 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf4142f50 con 0x7facf40b1280 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011800436 0 0) 0x7facf41ccab0 con 0x7facf409ac40 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.704+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf41422a0 con 0x7facf409ac40 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3567615212 0 0) 0x7facd8004eb0 con 0x7facf41d6ed0 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facf41ccab0 con 0x7facf41d6ed0 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7facec001c70 con 0x7facf41d6ed0 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:30.715 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient(hunting): dump: 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 953604902 0 0) 0x7facf4142f50 con 0x7facf40b1280 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facd8004eb0 con 0x7facf40b1280 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 143337673 0 0) 0x7facf41422a0 con 0x7facf409ac40 2024-03-29T11:34:30.716 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facf4142f50 con 0x7facf409ac40 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7face0001920 con 0x7facf40b1280 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient(hunting): dump: 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.717 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2451198604 0 0) 0x7facf41ccab0 con 0x7facf41d6ed0 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 >> v1:172.21.15.22:6790/0 conn(0x7facf409ac40 legacy=0x7facf4141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 >> v1:172.21.15.22:6789/0 conn(0x7facf40b1280 legacy=0x7facf4142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 monclient: found mon.b 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf41ccc40 con 0x7facf41d6ed0 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 10 monclient: _renew_subs 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.718 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf413b4d0 con 0x7facf41d6ed0 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 10 monclient: _renew_subs 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facf41e3720 con 0x7facf41d6ed0 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facfaa86640 1 librados: init done 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facec003990 con 0x7facf41d6ed0 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7facec003d20 con 0x7facf41d6ed0 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 monclient: dump: 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.719 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf2ffd640 4 set_mon_vals no callback set 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7facec004290 con 0x7facf41d6ed0 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1124920937 0 0) 0x7facec0068a0 con 0x7facf41d6ed0 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 10 client.4111.objecter ms_dispatch 0x7facf4000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.705+0000 7facf17fa640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T11:34:30.720 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.706+0000 7facfaa86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.706+0000 7facfaa86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.706+0000 7facfaa86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.706+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd4005080 con 0x7facf41d6ed0 2024-03-29T11:34:30.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.708+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7facec004720 con 0x7facf41d6ed0 2024-03-29T11:34:30.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.708+0000 7facf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.721 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.708+0000 7facf17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:30.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.780+0000 7facfaa86640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8a925ac5-9ea5-4bd1-84cf-9415000e9422", "id": 0}] 2024-03-29T11:34:30.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.780+0000 7facfaa86640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8a925ac5-9ea5-4bd1-84cf-9415000e9422", "id": 0}] 2024-03-29T11:34:30.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.780+0000 7facfaa86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.780+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8a925ac5-9ea5-4bd1-84cf-9415000e9422", "id": 0} v 0) v1 -- 0x7facd40058b0 con 0x7facf41d6ed0 2024-03-29T11:34:30.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.785+0000 7facf17fa640 1 -- 172.21.15.204:0/219218093 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8a925ac5-9ea5-4bd1-84cf-9415000e9422", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2944348933 0 732707515) 0x7facec006b30 con 0x7facf41d6ed0 2024-03-29T11:34:30.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.785+0000 7facf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8a925ac5-9ea5-4bd1-84cf-9415000e9422", "id": 0}] 2024-03-29T11:34:30.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.785+0000 7facf17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:30.783 INFO:teuthology.orchestra.run.smithi204.stdout:0 2024-03-29T11:34:30.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 10 librados: watch_flush enter 2024-03-29T11:34:30.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 10 librados: watch_flush exit 2024-03-29T11:34:30.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T11:34:30.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 10 client.4111.objecter successfully canceled tick 2024-03-29T11:34:30.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 10 monclient: shutdown 2024-03-29T11:34:30.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:30.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 >> v1:172.21.15.204:6789/0 conn(0x7facf41d6ed0 legacy=0x7facf413adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 shutdown_connections 2024-03-29T11:34:30.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 >> 172.21.15.204:0/219218093 conn(0x7facf4066b00 msgr2=0x7facf4066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 shutdown_connections 2024-03-29T11:34:30.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 1 -- 172.21.15.204:0/219218093 wait complete. 2024-03-29T11:34:30.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.786+0000 7facfaa86640 1 librados: shutdown 2024-03-29T11:34:30.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 6bcbc966-5d5a-4edc-954f-b404f3090523 1 2024-03-29T11:34:30.892 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.895+0000 7f85c8063640 10 monclient: get_monmap_and_config 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.895+0000 7f85c8063640 10 monclient: build_initial_monmap 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.895+0000 7f85c8063640 10 monclient: monmap: 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:34:30.896591+0000 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:34:30.896591+0000 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:30.893 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.896+0000 7f85c8063640 1 Processor -- start 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.896+0000 7f85c8063640 1 -- start start 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.896+0000 7f85c8063640 10 monclient: init 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 10 monclient(hunting): picked mon.noname-a con 0x7f85c009ac40 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 10 monclient(hunting): picked mon.noname-c con 0x7f85c009c870 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 10 monclient(hunting): picked mon.noname-b con 0x7f85c00b4360 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:30.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c009dec0 con 0x7f85c009ac40 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c009da30 con 0x7f85c009c870 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00b39c0 con 0x7f85c00b4360 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 10 monclient(hunting): authenticate will time out at 907.163077s 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c55fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f85c009c870 0x7f85c013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49910/0 (socket says 172.21.15.204:49910) 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c55fe640 1 -- 172.21.15.204:0/3914704758 learned_addr learned my addr 172.21.15.204:0/3914704758 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197569000 0 0) 0x7f85c009da30 con 0x7f85c009c870 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ac003520 con 0x7f85c009c870 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760091584 0 0) 0x7f85bc001110 con 0x7f85c009ac40 2024-03-29T11:34:30.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c009da30 con 0x7f85c009ac40 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084653752 0 0) 0x7f85c00b39c0 con 0x7f85c00b4360 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85bc001110 con 0x7f85c00b4360 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2206054833 0 0) 0x7f85ac003520 con 0x7f85c009c870 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 >> v1:172.21.15.22:6790/0 conn(0x7f85c00b4360 legacy=0x7f85c014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 >> v1:172.21.15.22:6789/0 conn(0x7f85c009ac40 legacy=0x7f85c00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.903 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 monclient: found mon.noname-c 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00b5160 con 0x7f85c009c870 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: _finish_auth 0 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f85b0001690 con 0x7f85c009c870 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: dump: 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.904 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85b0001870 con 0x7f85c009c870 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 1 -- 172.21.15.204:0/3914704758 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f85b0001c80 con 0x7f85c009c870 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:30.905 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c8063640 4 set_mon_vals no callback set 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.897+0000 7f85c4dfd640 10 monclient: dump: 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.906 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.898+0000 7f85c8063640 10 monclient: shutdown 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.898+0000 7f85c8063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.898+0000 7f85c8063640 1 -- 172.21.15.204:0/3914704758 >> v1:172.21.15.204:6789/0 conn(0x7f85c009c870 legacy=0x7f85c013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.898+0000 7f85c8063640 1 -- 172.21.15.204:0/3914704758 shutdown_connections 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.898+0000 7f85c8063640 1 -- 172.21.15.204:0/3914704758 >> 172.21.15.204:0/3914704758 conn(0x7f85c0066b00 msgr2=0x7f85c0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.898+0000 7f85c8063640 1 -- 172.21.15.204:0/3914704758 shutdown_connections 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.898+0000 7f85c8063640 1 -- 172.21.15.204:0/3914704758 wait complete. 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 10 monclient: build_initial_monmap 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 10 monclient: monmap: 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.907 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 1 librados: starting msgr at 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 1 librados: starting objecter 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 1 Processor -- start 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 1 -- start start 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 1 librados: setting wanted keys 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 1 librados: calling monclient init 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.899+0000 7f85c8063640 10 monclient: init 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 10 monclient(hunting): picked mon.noname-c con 0x7f85c009ac40 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 10 monclient(hunting): picked mon.noname-a con 0x7f85c01d7bc0 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:30.908 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 10 monclient(hunting): picked mon.noname-b con 0x7f85c00b17f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c013b890 con 0x7f85c01d7bc0 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c01d76a0 con 0x7f85c00b17f0 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c01406b0 con 0x7f85c009ac40 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c8063640 10 monclient(hunting): authenticate will time out at 907.166216s 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c6600640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f85c00b17f0 0x7f85c013b180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49916/0 (socket says 172.21.15.204:49916) 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85c6600640 1 -- 172.21.15.204:0/4127843688 learned_addr learned my addr 172.21.15.204:0/4127843688 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 client.?.objecter ms_handle_connect 0x7f85c00b17f0 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 client.?.objecter ms_handle_connect 0x7f85c009ac40 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 client.?.objecter ms_handle_connect 0x7f85c01d7bc0 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710125435 0 0) 0x7f85c01d76a0 con 0x7f85c00b17f0 2024-03-29T11:34:30.909 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a4004eb0 con 0x7f85c00b17f0 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442580625 0 0) 0x7f85c013b890 con 0x7f85c01d7bc0 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c01d76a0 con 0x7f85c01d7bc0 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765065008 0 0) 0x7f85c01406b0 con 0x7f85c009ac40 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:30.910 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c013b890 con 0x7f85c009ac40 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230983693 0 0) 0x7f85a4004eb0 con 0x7f85c00b17f0 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85c01406b0 con 0x7f85c00b17f0 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f85b8001c10 con 0x7f85c00b17f0 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): dump: 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.911 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 771400138 0 0) 0x7f85c01d76a0 con 0x7f85c01d7bc0 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85a4004eb0 con 0x7f85c01d7bc0 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f85b00019e0 con 0x7f85c01d7bc0 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): dump: 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.912 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3264031042 0 0) 0x7f85c013b890 con 0x7f85c009ac40 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85c01d76a0 con 0x7f85c009ac40 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f85bc001860 con 0x7f85c009ac40 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.900+0000 7f85b67fc640 10 monclient(hunting): dump: 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.913 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2384877289 0 0) 0x7f85c01406b0 con 0x7f85c00b17f0 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 >> v1:172.21.15.22:6790/0 conn(0x7f85c009ac40 legacy=0x7f85c013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 >> v1:172.21.15.22:6789/0 conn(0x7f85c01d7bc0 legacy=0x7f85c013aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 monclient: found mon.b 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c0140840 con 0x7f85c00b17f0 2024-03-29T11:34:30.914 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 10 monclient: _renew_subs 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c00b6f40 con 0x7f85c00b17f0 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 10 monclient: _renew_subs 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c01d6f40 con 0x7f85c00b17f0 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85c8063640 1 librados: init done 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85b8003990 con 0x7f85c00b17f0 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:30.915 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f85b8002100 con 0x7f85c00b17f0 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b7fff640 4 set_mon_vals no callback set 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 monclient: dump: 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:30.916 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f85b80041b0 con 0x7f85c00b17f0 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1838756644 0 0) 0x7f85b8006930 con 0x7f85c00b17f0 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 10 client.4117.objecter ms_dispatch 0x7f85c0000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.901+0000 7f85b67fc640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.902+0000 7f85c8063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.902+0000 7f85c8063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.902+0000 7f85c8063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.902+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a8005080 con 0x7f85c00b17f0 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.904+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f85b8004690 con 0x7f85c00b17f0 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.904+0000 7f85b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:30.917 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.904+0000 7f85b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:30.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.976+0000 7f85c8063640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6bcbc966-5d5a-4edc-954f-b404f3090523", "id": 1}] 2024-03-29T11:34:30.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.976+0000 7f85c8063640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6bcbc966-5d5a-4edc-954f-b404f3090523", "id": 1}] 2024-03-29T11:34:30.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.976+0000 7f85c8063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:30.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.976+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6bcbc966-5d5a-4edc-954f-b404f3090523", "id": 1} v 0) v1 -- 0x7f85a80058b0 con 0x7f85c00b17f0 2024-03-29T11:34:30.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.984+0000 7f85b67fc640 1 -- 172.21.15.204:0/4127843688 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6bcbc966-5d5a-4edc-954f-b404f3090523", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2666969954 0 940483276) 0x7f85b8005b40 con 0x7f85c00b17f0 2024-03-29T11:34:30.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.984+0000 7f85b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6bcbc966-5d5a-4edc-954f-b404f3090523", "id": 1}] 2024-03-29T11:34:30.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.984+0000 7f85b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:30.981 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-29T11:34:30.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 10 librados: watch_flush enter 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 10 librados: watch_flush exit 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 10 monclient: shutdown 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 >> v1:172.21.15.204:6789/0 conn(0x7f85c00b17f0 legacy=0x7f85c013b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 shutdown_connections 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 >> 172.21.15.204:0/4127843688 conn(0x7f85c0066b00 msgr2=0x7f85c013d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 shutdown_connections 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 1 -- 172.21.15.204:0/4127843688 wait complete. 2024-03-29T11:34:30.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:30.985+0000 7f85c8063640 1 librados: shutdown 2024-03-29T11:34:30.992 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 99bf3692-4e61-4936-b8ec-082a3bd7b4d4 2 2024-03-29T11:34:31.091 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.093+0000 7f11d3233640 10 monclient: get_monmap_and_config 2024-03-29T11:34:31.091 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.093+0000 7f11d3233640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.091 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.093+0000 7f11d3233640 10 monclient: monmap: 2024-03-29T11:34:31.091 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.091 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.091 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:34:31.094806+0000 2024-03-29T11:34:31.091 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:34:31.094806+0000 2024-03-29T11:34:31.091 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.092 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.092 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.092 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:31.092 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:31.092 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.103 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.094+0000 7f11d3233640 1 Processor -- start 2024-03-29T11:34:31.103 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.094+0000 7f11d3233640 1 -- start start 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.094+0000 7f11d3233640 10 monclient: init 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 10 monclient(hunting): picked mon.noname-a con 0x7f11cc09ac40 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 10 monclient(hunting): picked mon.noname-c con 0x7f11cc09c870 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 10 monclient(hunting): picked mon.noname-b con 0x7f11cc0b4360 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc09dec0 con 0x7f11cc09ac40 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc09da30 con 0x7f11cc09c870 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc0b39c0 con 0x7f11cc0b4360 2024-03-29T11:34:31.104 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 10 monclient(hunting): authenticate will time out at 907.361034s 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cbfff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f11cc09c870 0x7f11cc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49928/0 (socket says 172.21.15.204:49928) 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cbfff640 1 -- 172.21.15.204:0/1758059773 learned_addr learned my addr 172.21.15.204:0/1758059773 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725482894 0 0) 0x7f11cc09da30 con 0x7f11cc09c870 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:34:31.105 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11bc003520 con 0x7f11cc09c870 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814523862 0 0) 0x7f11c0001110 con 0x7f11cc09ac40 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11cc09da30 con 0x7f11cc09ac40 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498144618 0 0) 0x7f11cc0b39c0 con 0x7f11cc0b4360 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.106 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c0001110 con 0x7f11cc0b4360 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705774888 0 0) 0x7f11bc003520 con 0x7f11cc09c870 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 >> v1:172.21.15.22:6790/0 conn(0x7f11cc0b4360 legacy=0x7f11cc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 >> v1:172.21.15.22:6789/0 conn(0x7f11cc09ac40 legacy=0x7f11cc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc0b5160 con 0x7f11cc09c870 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.107 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f11b8001690 con 0x7f11cc09c870 2024-03-29T11:34:31.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:31.108 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: dump: 2024-03-29T11:34:31.108 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.108 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b80017f0 con 0x7f11cc09c870 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 1 -- 172.21.15.204:0/1758059773 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f11b8001c00 con 0x7f11cc09c870 2024-03-29T11:34:31.109 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11cb7fe640 10 monclient: dump: 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.110 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.095+0000 7f11d3233640 4 set_mon_vals no callback set 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.096+0000 7f11d3233640 10 monclient: shutdown 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.096+0000 7f11d3233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.096+0000 7f11d3233640 1 -- 172.21.15.204:0/1758059773 >> v1:172.21.15.204:6789/0 conn(0x7f11cc09c870 legacy=0x7f11cc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.096+0000 7f11d3233640 1 -- 172.21.15.204:0/1758059773 shutdown_connections 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.096+0000 7f11d3233640 1 -- 172.21.15.204:0/1758059773 >> 172.21.15.204:0/1758059773 conn(0x7f11cc066b00 msgr2=0x7f11cc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.096+0000 7f11d3233640 1 -- 172.21.15.204:0/1758059773 shutdown_connections 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.096+0000 7f11d3233640 1 -- 172.21.15.204:0/1758059773 wait complete. 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 10 monclient: monmap: 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-29T11:34:31.111 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 1 librados: starting msgr at 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 1 librados: starting objecter 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 1 Processor -- start 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 1 -- start start 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 1 librados: setting wanted keys 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 1 librados: calling monclient init 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.097+0000 7f11d3233640 10 monclient: init 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:31.112 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 10 monclient(hunting): picked mon.noname-a con 0x7f11cc09ac40 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 10 monclient(hunting): picked mon.noname-b con 0x7f11cc13ffa0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 10 monclient(hunting): picked mon.noname-c con 0x7f11cc1409b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc13a7f0 con 0x7f11cc09ac40 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc140600 con 0x7f11cc13ffa0 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc1d78d0 con 0x7f11cc1409b0 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11d3233640 10 monclient(hunting): authenticate will time out at 907.364107s 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11cbfff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f11cc13ffa0 0x7f11cc1399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49930/0 (socket says 172.21.15.204:49930) 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11cbfff640 1 -- 172.21.15.204:0/3516112487 learned_addr learned my addr 172.21.15.204:0/3516112487 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 client.?.objecter ms_handle_connect 0x7f11cc13ffa0 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 client.?.objecter ms_handle_connect 0x7f11cc09ac40 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.113 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 client.?.objecter ms_handle_connect 0x7f11cc1409b0 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840210786 0 0) 0x7f11cc140600 con 0x7f11cc13ffa0 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b0004eb0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545771561 0 0) 0x7f11cc13a7f0 con 0x7f11cc09ac40 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11cc140600 con 0x7f11cc09ac40 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932761234 0 0) 0x7f11cc1d78d0 con 0x7f11cc1409b0 2024-03-29T11:34:31.114 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11cc13a7f0 con 0x7f11cc1409b0 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180739175 0 0) 0x7f11b0004eb0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11cc1d78d0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f11b8001bc0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): dump: 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.115 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 222368496 0 0) 0x7f11cc140600 con 0x7f11cc09ac40 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11b0004eb0 con 0x7f11cc09ac40 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f11c00018e0 con 0x7f11cc09ac40 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): dump: 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.116 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976670212 0 0) 0x7f11cc13a7f0 con 0x7f11cc1409b0 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11cc140600 con 0x7f11cc1409b0 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f11c4001bd0 con 0x7f11cc1409b0 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 10 monclient(hunting): dump: 2024-03-29T11:34:31.117 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4270878425 0 0) 0x7f11cc1d78d0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 >> v1:172.21.15.22:6790/0 conn(0x7f11cc1409b0 legacy=0x7f11cc13a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.098+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 >> v1:172.21.15.22:6789/0 conn(0x7f11cc09ac40 legacy=0x7f11cc1d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 1 monclient: found mon.b 2024-03-29T11:34:31.118 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc1427d0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 10 monclient: _renew_subs 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11cc142a80 con 0x7f11cc13ffa0 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 10 monclient: _renew_subs 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc0b80a0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.119 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11d3233640 1 librados: init done 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b80011f0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f11b8004c30 con 0x7f11cc13ffa0 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 monclient: dump: 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.120 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11caffd640 4 set_mon_vals no callback set 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f11b80051e0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1111709444 0 0) 0x7f11b8006860 con 0x7f11cc13ffa0 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 10 client.4123.objecter ms_dispatch 0x7f11cc000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.099+0000 7f11c97fa640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.100+0000 7f11d3233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.100+0000 7f11d3233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.100+0000 7f11d3233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.100+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac005080 con 0x7f11cc13ffa0 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.102+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f11b8006410 con 0x7f11cc13ffa0 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.102+0000 7f11c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.121 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.102+0000 7f11c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:31.122 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.104+0000 7f3548438040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:31.122 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.105+0000 7f062ea3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:34:31.158 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.161+0000 7f062ea3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:34:31.158 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.161+0000 7f3548438040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:34:31.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.174+0000 7f11d3233640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "99bf3692-4e61-4936-b8ec-082a3bd7b4d4", "id": 2}] 2024-03-29T11:34:31.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.174+0000 7f11d3233640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "99bf3692-4e61-4936-b8ec-082a3bd7b4d4", "id": 2}] 2024-03-29T11:34:31.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.174+0000 7f11d3233640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.174+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd new", "uuid": "99bf3692-4e61-4936-b8ec-082a3bd7b4d4", "id": 2} v 0) v1 -- 0x7f11ac0058b0 con 0x7f11cc13ffa0 2024-03-29T11:34:31.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.178+0000 7f11c97fa640 1 -- 172.21.15.204:0/3516112487 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "99bf3692-4e61-4936-b8ec-082a3bd7b4d4", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3999369200 0 216599125) 0x7f11b8006a20 con 0x7f11cc13ffa0 2024-03-29T11:34:31.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.178+0000 7f11c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "99bf3692-4e61-4936-b8ec-082a3bd7b4d4", "id": 2}] 2024-03-29T11:34:31.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.178+0000 7f11c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:31.175 INFO:teuthology.orchestra.run.smithi204.stdout:2 2024-03-29T11:34:31.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 10 librados: watch_flush enter 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 10 librados: watch_flush exit 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 10 monclient: shutdown 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 >> v1:172.21.15.204:6789/0 conn(0x7f11cc13ffa0 legacy=0x7f11cc1399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 shutdown_connections 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 >> 172.21.15.204:0/3516112487 conn(0x7f11cc066b00 msgr2=0x7f11cc13e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 shutdown_connections 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 1 -- 172.21.15.204:0/3516112487 wait complete. 2024-03-29T11:34:31.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.179+0000 7f11d3233640 1 librados: shutdown 2024-03-29T11:34:31.186 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 8de7905a-e6f7-43ab-896e-4c9dddb96521 3 2024-03-29T11:34:31.205 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.208+0000 7f062ea3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:31.207 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.210+0000 7f3548438040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:34:31.260 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.264+0000 7f062ea3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:31.265 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.268+0000 7f3548438040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.289+0000 7f5435206640 10 monclient: get_monmap_and_config 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.289+0000 7f5435206640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.289+0000 7f5435206640 10 monclient: monmap: 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:34:31.290194+0000 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:34:31.290194+0000 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.287 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.287 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:31.287 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:31.287 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.289+0000 7f5435206640 1 Processor -- start 2024-03-29T11:34:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.289+0000 7f5435206640 1 -- start start 2024-03-29T11:34:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.289+0000 7f5435206640 10 monclient: init 2024-03-29T11:34:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 10 monclient(hunting): picked mon.noname-c con 0x7f543009ac40 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 10 monclient(hunting): picked mon.noname-b con 0x7f543009c870 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 10 monclient(hunting): picked mon.noname-a con 0x7f54300b4360 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543009dec0 con 0x7f54300b4360 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543009da30 con 0x7f543009ac40 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54300b39c0 con 0x7f543009c870 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f5435206640 10 monclient(hunting): authenticate will time out at 907.556484s 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f542ed9d640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f543009ac40 0x7f54300b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49938/0 (socket says 172.21.15.204:49938) 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f542ed9d640 1 -- 172.21.15.204:0/2749000398 learned_addr learned my addr 172.21.15.204:0/2749000398 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618965695 0 0) 0x7f543009da30 con 0x7f543009ac40 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f542e59c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f542e59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f542e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f542e59c640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.290+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f541c003520 con 0x7f543009ac40 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199349052 0 0) 0x7f5428001490 con 0x7f54300b4360 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543009da30 con 0x7f54300b4360 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201722439 0 0) 0x7f54300b39c0 con 0x7f543009c870 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5428001490 con 0x7f543009c870 2024-03-29T11:34:31.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 412872627 0 0) 0x7f541c003520 con 0x7f543009ac40 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 >> v1:172.21.15.22:6790/0 conn(0x7f543009c870 legacy=0x7f543013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 >> v1:172.21.15.22:6789/0 conn(0x7f54300b4360 legacy=0x7f543014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 monclient: found mon.noname-c 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54300b5160 con 0x7f543009ac40 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5420001690 con 0x7f543009ac40 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: dump: 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5420001870 con 0x7f543009ac40 2024-03-29T11:34:31.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 1 -- 172.21.15.204:0/2749000398 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5420001c80 con 0x7f543009ac40 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 4 set_mon_vals no callback set 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f542e59c640 10 monclient: dump: 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.300 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 10 monclient: shutdown 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 1 -- 172.21.15.204:0/2749000398 >> v1:172.21.15.204:6789/0 conn(0x7f543009ac40 legacy=0x7f54300b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 1 -- 172.21.15.204:0/2749000398 shutdown_connections 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.291+0000 7f5435206640 1 -- 172.21.15.204:0/2749000398 >> 172.21.15.204:0/2749000398 conn(0x7f5430066b00 msgr2=0x7f5430066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.292+0000 7f5435206640 1 -- 172.21.15.204:0/2749000398 shutdown_connections 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.292+0000 7f5435206640 1 -- 172.21.15.204:0/2749000398 wait complete. 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.292+0000 7f5435206640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.292+0000 7f5435206640 10 monclient: monmap: 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.301 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 librados: starting msgr at 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 librados: starting objecter 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 Processor -- start 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 -- start start 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 librados: setting wanted keys 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 librados: calling monclient init 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 10 monclient: init 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 10 monclient(hunting): picked mon.noname-b con 0x7f543009ac40 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 10 monclient(hunting): picked mon.noname-a con 0x7f54300b0210 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 10 monclient(hunting): picked mon.noname-c con 0x7f54301d7c90 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54300afb50 con 0x7f54300b0210 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54301d70f0 con 0x7f543009ac40 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54300b0650 con 0x7f54301d7c90 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f5435206640 10 monclient(hunting): authenticate will time out at 907.559605s 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f542ed9d640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f543009ac40 0x7f54300af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49944/0 (socket says 172.21.15.204:49944) 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f542ed9d640 1 -- 172.21.15.204:0/286487280 learned_addr learned my addr 172.21.15.204:0/286487280 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f54277fe640 10 client.?.objecter ms_handle_connect 0x7f543009ac40 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f54277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f54277fe640 10 client.?.objecter ms_handle_connect 0x7f54301d7c90 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f54277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f54277fe640 10 client.?.objecter ms_handle_connect 0x7f54300b0210 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f54277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.293+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958777788 0 0) 0x7f54301d70f0 con 0x7f543009ac40 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5410004eb0 con 0x7f543009ac40 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446042383 0 0) 0x7f54300b0650 con 0x7f54301d7c90 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54301d70f0 con 0x7f54301d7c90 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70087576 0 0) 0x7f54300afb50 con 0x7f54300b0210 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54300b0650 con 0x7f54300b0210 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1350791760 0 0) 0x7f5410004eb0 con 0x7f543009ac40 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54300afb50 con 0x7f543009ac40 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5420001bc0 con 0x7f543009ac40 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): dump: 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.305 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3250287257 0 0) 0x7f54300b0650 con 0x7f54300b0210 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5410004eb0 con 0x7f54300b0210 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f54180017e0 con 0x7f54300b0210 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): dump: 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.306 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1075286004 0 0) 0x7f54301d70f0 con 0x7f54301d7c90 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54300b0650 con 0x7f54301d7c90 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5428001a00 con 0x7f54301d7c90 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient(hunting): dump: 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.307 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3631662439 0 0) 0x7f54300afb50 con 0x7f543009ac40 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 >> v1:172.21.15.22:6790/0 conn(0x7f54301d7c90 legacy=0x7f54301e1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 >> v1:172.21.15.22:6789/0 conn(0x7f54300b0210 legacy=0x7f54301d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 monclient: found mon.b 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54300acd00 con 0x7f543009ac40 2024-03-29T11:34:31.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 10 monclient: _renew_subs 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54300b18a0 con 0x7f543009ac40 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 10 monclient: _renew_subs 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54301e35d0 con 0x7f543009ac40 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f5435206640 1 librados: init done 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5420004bf0 con 0x7f543009ac40 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5420004f00 con 0x7f543009ac40 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 monclient: dump: 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f542dd9b640 4 set_mon_vals no callback set 2024-03-29T11:34:31.310 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f54200053f0 con 0x7f543009ac40 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4149413522 0 0) 0x7f5420006a10 con 0x7f543009ac40 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 10 client.4129.objecter ms_dispatch 0x7f5430000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.294+0000 7f54277fe640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.295+0000 7f5435206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.295+0000 7f5435206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.295+0000 7f5435206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.295+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5414005080 con 0x7f543009ac40 2024-03-29T11:34:31.311 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.299+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5420006630 con 0x7f543009ac40 2024-03-29T11:34:31.312 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.299+0000 7f54277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.312 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.299+0000 7f54277fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:31.312 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.309+0000 7f062ea3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:31.312 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.315+0000 7f3548438040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:34:31.369 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.372+0000 7f5435206640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8de7905a-e6f7-43ab-896e-4c9dddb96521", "id": 3}] 2024-03-29T11:34:31.369 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.372+0000 7f5435206640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8de7905a-e6f7-43ab-896e-4c9dddb96521", "id": 3}] 2024-03-29T11:34:31.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.372+0000 7f5435206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.372+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8de7905a-e6f7-43ab-896e-4c9dddb96521", "id": 3} v 0) v1 -- 0x7f54140058b0 con 0x7f543009ac40 2024-03-29T11:34:31.374 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.376+0000 7f54277fe640 1 -- 172.21.15.204:0/286487280 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8de7905a-e6f7-43ab-896e-4c9dddb96521", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 635411438 0 525046306) 0x7f5420005830 con 0x7f543009ac40 2024-03-29T11:34:31.374 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.376+0000 7f54277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8de7905a-e6f7-43ab-896e-4c9dddb96521", "id": 3}] 2024-03-29T11:34:31.374 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.376+0000 7f54277fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:31.374 INFO:teuthology.orchestra.run.smithi204.stdout:3 2024-03-29T11:34:31.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 10 librados: watch_flush enter 2024-03-29T11:34:31.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 10 librados: watch_flush exit 2024-03-29T11:34:31.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 10 monclient: shutdown 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 >> v1:172.21.15.204:6789/0 conn(0x7f543009ac40 legacy=0x7f54300af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 shutdown_connections 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 >> 172.21.15.204:0/286487280 conn(0x7f5430066b00 msgr2=0x7f54300b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 shutdown_connections 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.377+0000 7f5435206640 1 -- 172.21.15.204:0/286487280 wait complete. 2024-03-29T11:34:31.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.378+0000 7f5435206640 1 librados: shutdown 2024-03-29T11:34:31.385 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new b730bd7f-7744-45f5-979d-4fbc1904ad26 4 2024-03-29T11:34:31.426 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.430+0000 7f062ea3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:31.434 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.437+0000 7f3548438040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:34:31.499 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.502+0000 7f062ea3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:31.509 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.512+0000 7f3548438040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:31.525 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.528+0000 7fe32f940640 10 monclient: get_monmap_and_config 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.528+0000 7fe32f940640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.528+0000 7fe32f940640 10 monclient: monmap: 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:34:31.529735+0000 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:34:31.529735+0000 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:31.526 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.529+0000 7fe32f940640 1 Processor -- start 2024-03-29T11:34:31.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.529+0000 7fe32f940640 1 -- start start 2024-03-29T11:34:31.535 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.529+0000 7fe32f940640 10 monclient: init 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 10 monclient(hunting): picked mon.noname-b con 0x7fe32809ac40 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 10 monclient(hunting): picked mon.noname-c con 0x7fe32809c870 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 10 monclient(hunting): picked mon.noname-a con 0x7fe3280b4360 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32809dec0 con 0x7fe3280b4360 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32809da30 con 0x7fe32809c870 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3280b39c0 con 0x7fe32809ac40 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 10 monclient(hunting): authenticate will time out at 907.795968s 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32cedb640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fe32809c870 0x7fe32813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49950/0 (socket says 172.21.15.204:49950) 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32cedb640 1 -- 172.21.15.204:0/456152208 learned_addr learned my addr 172.21.15.204:0/456152208 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347756111 0 0) 0x7fe32809da30 con 0x7fe32809c870 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.536 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314003520 con 0x7fe32809c870 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271366314 0 0) 0x7fe320001490 con 0x7fe3280b4360 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe32809da30 con 0x7fe3280b4360 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761473199 0 0) 0x7fe3280b39c0 con 0x7fe32809ac40 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:34:31.537 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe320001490 con 0x7fe32809ac40 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397735119 0 0) 0x7fe32809da30 con 0x7fe3280b4360 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 >> v1:172.21.15.22:6790/0 conn(0x7fe32809ac40 legacy=0x7fe3280b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 >> v1:172.21.15.204:6789/0 conn(0x7fe32809c870 legacy=0x7fe32813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 monclient: found mon.noname-a 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3280b5160 con 0x7fe3280b4360 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 5 monclient: authenticate success, global_id 4137 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe3200019e0 con 0x7fe3280b4360 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.538 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: dump: 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.539 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe320001b40 con 0x7fe3280b4360 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 1 -- 172.21.15.204:0/456152208 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe320001f90 con 0x7fe3280b4360 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe32f940640 4 set_mon_vals no callback set 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.530+0000 7fe31ffff640 10 monclient: dump: 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.540 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.531+0000 7fe32f940640 10 monclient: shutdown 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.531+0000 7fe32f940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.531+0000 7fe32f940640 1 -- 172.21.15.204:0/456152208 >> v1:172.21.15.22:6789/0 conn(0x7fe3280b4360 legacy=0x7fe32814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.531+0000 7fe32f940640 1 -- 172.21.15.204:0/456152208 shutdown_connections 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.531+0000 7fe32f940640 1 -- 172.21.15.204:0/456152208 >> 172.21.15.204:0/456152208 conn(0x7fe328066b00 msgr2=0x7fe328066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.531+0000 7fe32f940640 1 -- 172.21.15.204:0/456152208 shutdown_connections 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.531+0000 7fe32f940640 1 -- 172.21.15.204:0/456152208 wait complete. 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 10 monclient: monmap: 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.541 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 1 librados: starting msgr at 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 1 librados: starting objecter 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 1 Processor -- start 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 1 -- start start 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 1 librados: setting wanted keys 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 1 librados: calling monclient init 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.532+0000 7fe32f940640 10 monclient: init 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:31.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 10 monclient(hunting): picked mon.noname-c con 0x7fe32809ac40 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 10 monclient(hunting): picked mon.noname-a con 0x7fe328146060 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 10 monclient(hunting): picked mon.noname-b con 0x7fe3281da830 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe328141650 con 0x7fe328146060 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32813ffa0 con 0x7fe3281da830 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe328140130 con 0x7fe32809ac40 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32f940640 10 monclient(hunting): authenticate will time out at 907.799085s 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32dedd640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fe3281da830 0x7fe3281456d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49958/0 (socket says 172.21.15.204:49958) 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe32dedd640 1 -- 172.21.15.204:0/3258465648 learned_addr learned my addr 172.21.15.204:0/3258465648 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 client.?.objecter ms_handle_connect 0x7fe3281da830 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 client.?.objecter ms_handle_connect 0x7fe32809ac40 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 client.?.objecter ms_handle_connect 0x7fe328146060 2024-03-29T11:34:31.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985369584 0 0) 0x7fe32813ffa0 con 0x7fe3281da830 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe30c004eb0 con 0x7fe3281da830 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795224985 0 0) 0x7fe328140130 con 0x7fe32809ac40 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe32813ffa0 con 0x7fe32809ac40 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311540777 0 0) 0x7fe328141650 con 0x7fe328146060 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe328140130 con 0x7fe328146060 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2864722993 0 0) 0x7fe30c004eb0 con 0x7fe3281da830 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe328141650 con 0x7fe3281da830 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe320001520 con 0x7fe3281da830 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.545 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984072354 0 0) 0x7fe328140130 con 0x7fe328146060 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe30c004eb0 con 0x7fe328146060 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1945622244 0 0) 0x7fe32813ffa0 con 0x7fe32809ac40 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe328140130 con 0x7fe32809ac40 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe318001950 con 0x7fe328146060 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.546 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe324001910 con 0x7fe32809ac40 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.547 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.533+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1489843395 0 0) 0x7fe328141650 con 0x7fe3281da830 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 >> v1:172.21.15.22:6790/0 conn(0x7fe32809ac40 legacy=0x7fe3281da120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 >> v1:172.21.15.22:6789/0 conn(0x7fe328146060 legacy=0x7fe328144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 1 monclient: found mon.b 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3281402c0 con 0x7fe3281da830 2024-03-29T11:34:31.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 10 monclient: _renew_subs 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe328140ad0 con 0x7fe3281da830 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe320003c20 con 0x7fe3281da830 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe320003fb0 con 0x7fe3281da830 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 monclient: dump: 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.549 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 10 monclient: _renew_subs 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31f7fe640 4 set_mon_vals no callback set 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3281e34d0 con 0x7fe3281da830 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe32f940640 1 librados: init done 2024-03-29T11:34:31.550 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe320004520 con 0x7fe3281da830 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 441669082 0 0) 0x7fe320006010 con 0x7fe3281da830 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 10 client.4135.objecter ms_dispatch 0x7fe328000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.534+0000 7fe31dffb640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.535+0000 7fe32f940640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.535+0000 7fe32f940640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.535+0000 7fe32f940640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.535+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3281414a0 con 0x7fe3281da830 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.537+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe3200061d0 con 0x7fe3281da830 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.537+0000 7fe31dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.551 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.537+0000 7fe31dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:31.552 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.551+0000 7f062ea3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:31.559 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.562+0000 7f3548438040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:34:31.602 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.605+0000 7f062ea3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:31.606 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.609+0000 7fe32f940640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b730bd7f-7744-45f5-979d-4fbc1904ad26", "id": 4}] 2024-03-29T11:34:31.606 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.609+0000 7fe32f940640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b730bd7f-7744-45f5-979d-4fbc1904ad26", "id": 4}] 2024-03-29T11:34:31.606 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.609+0000 7fe32f940640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.606 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.609+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b730bd7f-7744-45f5-979d-4fbc1904ad26", "id": 4} v 0) v1 -- 0x7fe32809b020 con 0x7fe3281da830 2024-03-29T11:34:31.609 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.612+0000 7fe31dffb640 1 -- 172.21.15.204:0/3258465648 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b730bd7f-7744-45f5-979d-4fbc1904ad26", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1755878149 0 1697012583) 0x7fe320006370 con 0x7fe3281da830 2024-03-29T11:34:31.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.612+0000 7fe31dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b730bd7f-7744-45f5-979d-4fbc1904ad26", "id": 4}] 2024-03-29T11:34:31.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.612+0000 7fe31dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:31.610 INFO:teuthology.orchestra.run.smithi204.stdout:4 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 10 librados: watch_flush enter 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 10 librados: watch_flush exit 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 10 monclient: shutdown 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 >> v1:172.21.15.204:6789/0 conn(0x7fe3281da830 legacy=0x7fe3281456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 shutdown_connections 2024-03-29T11:34:31.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 >> 172.21.15.204:0/3258465648 conn(0x7fe328066b00 msgr2=0x7fe32813a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.612 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 shutdown_connections 2024-03-29T11:34:31.612 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.613+0000 7fe32f940640 1 -- 172.21.15.204:0/3258465648 wait complete. 2024-03-29T11:34:31.612 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.614+0000 7fe32f940640 1 librados: shutdown 2024-03-29T11:34:31.614 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.617+0000 7f3548438040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:34:31.621 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 91c34317-366b-402e-8cab-d9e594d0273b 5 2024-03-29T11:34:31.652 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:31.655+0000 7f062ea3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:31.664 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:31.667+0000 7f3548438040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:34:31.760 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.763+0000 7faaa5ded640 10 monclient: get_monmap_and_config 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.763+0000 7faaa5ded640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.763+0000 7faaa5ded640 10 monclient: monmap: 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:34:31.764557+0000 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:34:31.764557+0000 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.761 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:31.762 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 1 Processor -- start 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 1 -- start start 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 10 monclient: init 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 10 monclient(hunting): picked mon.noname-b con 0x7faaa009ac40 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 10 monclient(hunting): picked mon.noname-c con 0x7faaa009c870 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 10 monclient(hunting): picked mon.noname-a con 0x7faaa00b4360 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.764+0000 7faaa5ded640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa009dec0 con 0x7faaa00b4360 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa009da30 con 0x7faaa009c870 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa00b39c0 con 0x7faaa009ac40 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 10 monclient(hunting): authenticate will time out at 908.030892s 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9effd640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7faaa009c870 0x7faaa013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49970/0 (socket says 172.21.15.204:49970) 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9effd640 1 -- 172.21.15.204:0/3674718113 learned_addr learned my addr 172.21.15.204:0/3674718113 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298916232 0 0) 0x7faaa009da30 con 0x7faaa009c870 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:34:31.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa88003520 con 0x7faaa009c870 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504268969 0 0) 0x7faa98001490 con 0x7faaa00b4360 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa009da30 con 0x7faaa00b4360 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514718514 0 0) 0x7faaa00b39c0 con 0x7faaa009ac40 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa98001490 con 0x7faaa009ac40 2024-03-29T11:34:31.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 662302358 0 0) 0x7faa88003520 con 0x7faaa009c870 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 >> v1:172.21.15.22:6790/0 conn(0x7faaa009ac40 legacy=0x7faaa00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 >> v1:172.21.15.22:6789/0 conn(0x7faaa00b4360 legacy=0x7faaa014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa00b5160 con 0x7faaa009c870 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa90001690 con 0x7faaa009c870 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.773 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: dump: 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.774 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa900017f0 con 0x7faaa009c870 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 1 -- 172.21.15.204:0/3674718113 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa90001c40 con 0x7faaa009c870 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faaa5ded640 4 set_mon_vals no callback set 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.765+0000 7faa9e7fc640 10 monclient: dump: 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.775 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.766+0000 7faaa5ded640 10 monclient: shutdown 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.766+0000 7faaa5ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.766+0000 7faaa5ded640 1 -- 172.21.15.204:0/3674718113 >> v1:172.21.15.204:6789/0 conn(0x7faaa009c870 legacy=0x7faaa013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.766+0000 7faaa5ded640 1 -- 172.21.15.204:0/3674718113 shutdown_connections 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.766+0000 7faaa5ded640 1 -- 172.21.15.204:0/3674718113 >> 172.21.15.204:0/3674718113 conn(0x7faaa0066b00 msgr2=0x7faaa0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.766+0000 7faaa5ded640 1 -- 172.21.15.204:0/3674718113 shutdown_connections 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.766+0000 7faaa5ded640 1 -- 172.21.15.204:0/3674718113 wait complete. 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 10 monclient: monmap: 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.776 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 1 librados: starting msgr at 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 1 librados: starting objecter 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 1 Processor -- start 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 1 -- start start 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 1 librados: setting wanted keys 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 1 librados: calling monclient init 2024-03-29T11:34:31.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.767+0000 7faaa5ded640 10 monclient: init 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient(hunting): picked mon.noname-b con 0x7faaa009ac40 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient(hunting): picked mon.noname-c con 0x7faaa0141390 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient(hunting): picked mon.noname-a con 0x7faaa0145c10 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa0140bd0 con 0x7faaa0145c10 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa0140920 con 0x7faaa009ac40 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa01d7340 con 0x7faaa0141390 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient(hunting): authenticate will time out at 908.034029s 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa9f7fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7faaa009ac40 0x7faaa0140210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49974/0 (socket says 172.21.15.204:49974) 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa9f7fe640 1 -- 172.21.15.204:0/430356872 learned_addr learned my addr 172.21.15.204:0/430356872 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 client.?.objecter ms_handle_connect 0x7faaa009ac40 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 client.?.objecter ms_handle_connect 0x7faaa0145c10 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030049317 0 0) 0x7faaa0140920 con 0x7faaa009ac40 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c004eb0 con 0x7faaa009ac40 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 client.?.objecter ms_handle_connect 0x7faaa0141390 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698269221 0 0) 0x7faaa0140bd0 con 0x7faaa0145c10 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa0140920 con 0x7faaa0145c10 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305149451 0 0) 0x7faaa01d7340 con 0x7faaa0141390 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa0140bd0 con 0x7faaa0141390 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 193145289 0 0) 0x7faa8c004eb0 con 0x7faaa009ac40 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faaa01d7340 con 0x7faaa009ac40 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa94001af0 con 0x7faaa009ac40 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:31.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): dump: 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2997024710 0 0) 0x7faaa0140920 con 0x7faaa0145c10 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa8c004eb0 con 0x7faaa0145c10 2024-03-29T11:34:31.781 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa98001b30 con 0x7faaa0145c10 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): dump: 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3443493182 0 0) 0x7faaa0140bd0 con 0x7faaa0141390 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faaa0140920 con 0x7faaa0141390 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa90001650 con 0x7faaa0141390 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient(hunting): dump: 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.783 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2660759154 0 0) 0x7faaa01d7340 con 0x7faaa009ac40 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 >> v1:172.21.15.22:6790/0 conn(0x7faaa0141390 legacy=0x7faaa01d6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 >> v1:172.21.15.22:6789/0 conn(0x7faaa0145c10 legacy=0x7faaa0144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 monclient: found mon.b 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa01d74d0 con 0x7faaa009ac40 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faa87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient: _renew_subs 2024-03-29T11:34:31.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa0145580 con 0x7faaa009ac40 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient: _renew_subs 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa01e3480 con 0x7faaa009ac40 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.768+0000 7faaa5ded640 1 librados: init done 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa94001c50 con 0x7faaa009ac40 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa94003a40 con 0x7faaa009ac40 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 10 monclient: dump: 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.785 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa9dffb640 4 set_mon_vals no callback set 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faa94003ff0 con 0x7faaa009ac40 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2128854202 0 0) 0x7faa94006870 con 0x7faaa009ac40 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 10 client.4141.objecter ms_dispatch 0x7faaa0000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:34:31.786 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faa87fff640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faaa5ded640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faaa5ded640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faaa5ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.769+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa80005080 con 0x7faaa009ac40 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.771+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faa94006b50 con 0x7faaa009ac40 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.771+0000 7faa87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.771+0000 7faa87fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.842+0000 7faaa5ded640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "91c34317-366b-402e-8cab-d9e594d0273b", "id": 5}] 2024-03-29T11:34:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.842+0000 7faaa5ded640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "91c34317-366b-402e-8cab-d9e594d0273b", "id": 5}] 2024-03-29T11:34:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.842+0000 7faaa5ded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.842+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd new", "uuid": "91c34317-366b-402e-8cab-d9e594d0273b", "id": 5} v 0) v1 -- 0x7faa800058b0 con 0x7faaa009ac40 2024-03-29T11:34:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.845+0000 7faa87fff640 1 -- 172.21.15.204:0/430356872 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "91c34317-366b-402e-8cab-d9e594d0273b", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1823194499 0 1988412176) 0x7faa940063c0 con 0x7faaa009ac40 2024-03-29T11:34:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.845+0000 7faa87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "91c34317-366b-402e-8cab-d9e594d0273b", "id": 5}] 2024-03-29T11:34:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.845+0000 7faa87fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:31.843 INFO:teuthology.orchestra.run.smithi204.stdout:5 2024-03-29T11:34:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 10 librados: watch_flush enter 2024-03-29T11:34:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 10 librados: watch_flush exit 2024-03-29T11:34:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:34:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 10 monclient: shutdown 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 >> v1:172.21.15.204:6789/0 conn(0x7faaa009ac40 legacy=0x7faaa0140210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 shutdown_connections 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.846+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 >> 172.21.15.204:0/430356872 conn(0x7faaa0066b00 msgr2=0x7faaa01cb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.847+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 shutdown_connections 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.847+0000 7faaa5ded640 1 -- 172.21.15.204:0/430356872 wait complete. 2024-03-29T11:34:31.845 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.847+0000 7faaa5ded640 1 librados: shutdown 2024-03-29T11:34:31.854 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new c9df31e9-6a29-42fb-8314-2e7d0a11ced5 6 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.958+0000 7fb6732b3640 10 monclient: get_monmap_and_config 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.958+0000 7fb6732b3640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.958+0000 7fb6732b3640 10 monclient: monmap: 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:34:31.959693+0000 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:34:31.959693+0000 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:31.957 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:31.958 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.958 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.959+0000 7fb6732b3640 1 Processor -- start 2024-03-29T11:34:31.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.959+0000 7fb6732b3640 1 -- start start 2024-03-29T11:34:31.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.959+0000 7fb6732b3640 10 monclient: init 2024-03-29T11:34:31.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.959+0000 7fb6732b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.959+0000 7fb6732b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:31.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb66c09ac40 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb66c09c870 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb66c0b4360 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c09dec0 con 0x7fb66c09c870 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c09da30 con 0x7fb66c0b4360 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c0b39c0 con 0x7fb66c09ac40 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 10 monclient(hunting): authenticate will time out at 908.225962s 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb671850640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fb66c0b4360 0x7fb66c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:49990/0 (socket says 172.21.15.204:49990) 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb671850640 1 -- 172.21.15.204:0/3191730391 learned_addr learned my addr 172.21.15.204:0/3191730391 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685727488 0 0) 0x7fb66c09da30 con 0x7fb66c0b4360 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb658003520 con 0x7fb66c0b4360 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334090201 0 0) 0x7fb65c001040 con 0x7fb66c09c870 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb66c09da30 con 0x7fb66c09c870 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613443647 0 0) 0x7fb66c0b39c0 con 0x7fb66c09ac40 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65c001040 con 0x7fb66c09ac40 2024-03-29T11:34:31.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410900818 0 0) 0x7fb658003520 con 0x7fb66c0b4360 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 >> v1:172.21.15.22:6790/0 conn(0x7fb66c09ac40 legacy=0x7fb66c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 >> v1:172.21.15.22:6789/0 conn(0x7fb66c09c870 legacy=0x7fb66c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 monclient: found mon.noname-c 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0b5160 con 0x7fb66c0b4360 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb664001a10 con 0x7fb66c0b4360 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: dump: 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb664001bf0 con 0x7fb66c0b4360 2024-03-29T11:34:31.970 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 1 -- 172.21.15.204:0/3191730391 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb664002000 con 0x7fb66c0b4360 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb6732b3640 4 set_mon_vals no callback set 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.960+0000 7fb663fff640 10 monclient: dump: 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.971 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.961+0000 7fb6732b3640 10 monclient: shutdown 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.961+0000 7fb6732b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.961+0000 7fb6732b3640 1 -- 172.21.15.204:0/3191730391 >> v1:172.21.15.204:6789/0 conn(0x7fb66c0b4360 legacy=0x7fb66c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.961+0000 7fb6732b3640 1 -- 172.21.15.204:0/3191730391 shutdown_connections 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.961+0000 7fb6732b3640 1 -- 172.21.15.204:0/3191730391 >> 172.21.15.204:0/3191730391 conn(0x7fb66c066b00 msgr2=0x7fb66c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.961+0000 7fb6732b3640 1 -- 172.21.15.204:0/3191730391 shutdown_connections 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.961+0000 7fb6732b3640 1 -- 172.21.15.204:0/3191730391 wait complete. 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.962+0000 7fb6732b3640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.962+0000 7fb6732b3640 10 monclient: monmap: 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:31.972 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.962+0000 7fb6732b3640 1 librados: starting msgr at 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.962+0000 7fb6732b3640 1 librados: starting objecter 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.962+0000 7fb6732b3640 1 Processor -- start 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 1 -- start start 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 1 librados: setting wanted keys 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 1 librados: calling monclient init 2024-03-29T11:34:31.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 10 monclient: init 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb66c09ac40 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb66c1d7cd0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb66c0ad1e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c1d7130 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c139b10 con 0x7fb66c09ac40 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c13ffa0 con 0x7fb66c1d7cd0 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb6732b3640 10 monclient(hunting): authenticate will time out at 908.229840s 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.963+0000 7fb67104f640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fb66c09ac40 0x7fb66c1d75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:50000/0 (socket says 172.21.15.204:50000) 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb67104f640 1 -- 172.21.15.204:0/532923910 learned_addr learned my addr 172.21.15.204:0/532923910 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:31.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 client.?.objecter ms_handle_connect 0x7fb66c09ac40 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 client.?.objecter ms_handle_connect 0x7fb66c1d7cd0 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 client.?.objecter ms_handle_connect 0x7fb66c0ad1e0 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373151957 0 0) 0x7fb66c139b10 con 0x7fb66c09ac40 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb650004eb0 con 0x7fb66c09ac40 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721044109 0 0) 0x7fb66c1d7130 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.975 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb66c139b10 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697186619 0 0) 0x7fb66c13ffa0 con 0x7fb66c1d7cd0 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb66c1d7130 con 0x7fb66c1d7cd0 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1362183828 0 0) 0x7fb650004eb0 con 0x7fb66c09ac40 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb66c13ffa0 con 0x7fb66c09ac40 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb668001980 con 0x7fb66c09ac40 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:31.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2052994737 0 0) 0x7fb66c139b10 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb650004eb0 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb664001f00 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387903704 0 0) 0x7fb66c1d7130 con 0x7fb66c1d7cd0 2024-03-29T11:34:31.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb66c139b10 con 0x7fb66c1d7cd0 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb65c001850 con 0x7fb66c1d7cd0 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1453970962 0 0) 0x7fb650004eb0 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 >> v1:172.21.15.22:6790/0 conn(0x7fb66c1d7cd0 legacy=0x7fb66c138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 >> v1:172.21.15.204:6789/0 conn(0x7fb66c09ac40 legacy=0x7fb66c1d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 monclient: found mon.a 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0ae520 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb6732b3640 5 monclient: authenticate success, global_id 4152 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb6732b3640 10 monclient: _renew_subs 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb6732b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:31.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66c0ae800 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb664001460 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb664004fb0 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb661ffb640 10 monclient: dump: 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:31.981 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 10 monclient: _renew_subs 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.964+0000 7fb6637fe640 4 set_mon_vals no callback set 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c142b70 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb664005520 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 1 librados: init done 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1216937481 0 0) 0x7fb664005dc0 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb661ffb640 10 client.4152.objecter ms_dispatch 0x7fb66c000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb661ffb640 3 client.4152.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:34:31.982 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb661ffb640 3 client.4152.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:34:31.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb661ffb640 20 client.4152.objecter dump_active .. 0 homeless 2024-03-29T11:34:31.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:31.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.965+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66c1d80b0 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.968+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb664005f80 con 0x7fb66c0ad1e0 2024-03-29T11:34:31.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.968+0000 7fb661ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.983 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:31.968+0000 7fb661ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.042+0000 7fb6732b3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c9df31e9-6a29-42fb-8314-2e7d0a11ced5", "id": 6}] 2024-03-29T11:34:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.042+0000 7fb6732b3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c9df31e9-6a29-42fb-8314-2e7d0a11ced5", "id": 6}] 2024-03-29T11:34:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.042+0000 7fb6732b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.042+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c9df31e9-6a29-42fb-8314-2e7d0a11ced5", "id": 6} v 0) v1 -- 0x7fb66c04f730 con 0x7fb66c0ad1e0 2024-03-29T11:34:32.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.043+0000 7fb661ffb640 1 -- 172.21.15.204:0/532923910 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c9df31e9-6a29-42fb-8314-2e7d0a11ced5", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2834661040 0 1113811849) 0x7fb6640061f0 con 0x7fb66c0ad1e0 2024-03-29T11:34:32.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.043+0000 7fb661ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c9df31e9-6a29-42fb-8314-2e7d0a11ced5", "id": 6}] 2024-03-29T11:34:32.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.043+0000 7fb661ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:32.041 INFO:teuthology.orchestra.run.smithi204.stdout:6 2024-03-29T11:34:32.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.044+0000 7fb6732b3640 10 librados: watch_flush enter 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.044+0000 7fb6732b3640 10 librados: watch_flush exit 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.044+0000 7fb6732b3640 20 client.4152.objecter shutdown clearing up homeless session... 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.044+0000 7fb6732b3640 10 client.4152.objecter successfully canceled tick 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.044+0000 7fb6732b3640 10 monclient: shutdown 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.044+0000 7fb6732b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.044+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 >> v1:172.21.15.22:6789/0 conn(0x7fb66c0ad1e0 legacy=0x7fb66c139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.045+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 shutdown_connections 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.045+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 >> 172.21.15.204:0/532923910 conn(0x7fb66c066b00 msgr2=0x7fb66c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.045+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 shutdown_connections 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.045+0000 7fb6732b3640 1 -- 172.21.15.204:0/532923910 wait complete. 2024-03-29T11:34:32.043 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.045+0000 7fb6732b3640 1 librados: shutdown 2024-03-29T11:34:32.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new f54e98c9-12b9-4c5b-8939-a54cafe12e72 7 2024-03-29T11:34:32.055 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.058+0000 7f062ea3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:34:32.076 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.079+0000 7f3548438040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:34:32.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.155+0000 7faec516b640 10 monclient: get_monmap_and_config 2024-03-29T11:34:32.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.155+0000 7faec516b640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.152 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.155+0000 7faec516b640 10 monclient: monmap: 2024-03-29T11:34:32.152 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:32.152 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.152 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:34:32.156122+0000 2024-03-29T11:34:32.153 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:34:32.156122+0000 2024-03-29T11:34:32.153 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.153 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:32.153 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:32.153 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:32.153 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:32.153 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:32.162 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.155+0000 7faec516b640 1 Processor -- start 2024-03-29T11:34:32.162 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.155+0000 7faec516b640 1 -- start start 2024-03-29T11:34:32.162 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.155+0000 7faec516b640 10 monclient: init 2024-03-29T11:34:32.162 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.162 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:32.162 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 10 monclient(hunting): picked mon.noname-a con 0x7faec00b8240 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 10 monclient(hunting): picked mon.noname-c con 0x7faec00b1f60 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 10 monclient(hunting): picked mon.noname-b con 0x7faec009c8c0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec009bd80 con 0x7faec00b8240 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec009b960 con 0x7faec00b1f60 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec00b2ec0 con 0x7faec009c8c0 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faec516b640 10 monclient(hunting): authenticate will time out at 908.422313s 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebe59c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7faec00b1f60 0x7faec009c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:50014/0 (socket says 172.21.15.204:50014) 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebe59c640 1 -- 172.21.15.204:0/4238562294 learned_addr learned my addr 172.21.15.204:0/4238562294 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103128006 0 0) 0x7faec009b960 con 0x7faec00b1f60 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:34:32.163 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea8003520 con 0x7faec00b1f60 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511057380 0 0) 0x7faec009bd80 con 0x7faec00b8240 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec009b960 con 0x7faec00b8240 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852636237 0 0) 0x7faec00b2ec0 con 0x7faec009c8c0 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec009bd80 con 0x7faec009c8c0 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.156+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3306250819 0 0) 0x7faea8003520 con 0x7faec00b1f60 2024-03-29T11:34:32.164 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 >> v1:172.21.15.22:6790/0 conn(0x7faec009c8c0 legacy=0x7faec014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 >> v1:172.21.15.22:6789/0 conn(0x7faec00b8240 legacy=0x7faec013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 1 monclient: found mon.noname-c 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec013fb60 con 0x7faec00b1f60 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faeb4001690 con 0x7faec00b1f60 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: dump: 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:32.165 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faeb4001870 con 0x7faec00b1f60 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 1 -- 172.21.15.204:0/4238562294 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faeb4001c80 con 0x7faec00b1f60 2024-03-29T11:34:32.166 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 4 set_mon_vals no callback set 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faebdd9b640 10 monclient: dump: 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 10 monclient: shutdown 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.167 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 1 -- 172.21.15.204:0/4238562294 >> v1:172.21.15.204:6789/0 conn(0x7faec00b1f60 legacy=0x7faec009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 1 -- 172.21.15.204:0/4238562294 shutdown_connections 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 1 -- 172.21.15.204:0/4238562294 >> 172.21.15.204:0/4238562294 conn(0x7faec0066b00 msgr2=0x7faec0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 1 -- 172.21.15.204:0/4238562294 shutdown_connections 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.157+0000 7faec516b640 1 -- 172.21.15.204:0/4238562294 wait complete. 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.158+0000 7faec516b640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.158+0000 7faec516b640 10 monclient: monmap: 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:32.168 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.158+0000 7faec516b640 1 librados: starting msgr at 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.158+0000 7faec516b640 1 librados: starting objecter 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 1 Processor -- start 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 1 -- start start 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 1 librados: setting wanted keys 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 1 librados: calling monclient init 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 10 monclient: init 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 10 monclient(hunting): picked mon.noname-b con 0x7faec009c8c0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 10 monclient(hunting): picked mon.noname-a con 0x7faec00b8240 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 10 monclient(hunting): picked mon.noname-c con 0x7faec0132da0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec01d7350 con 0x7faec00b8240 2024-03-29T11:34:32.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec0133290 con 0x7faec009c8c0 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec0145090 con 0x7faec0132da0 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faec516b640 10 monclient(hunting): authenticate will time out at 908.425558s 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faebed9d640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7faec009c8c0 0x7faec0132690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:50024/0 (socket says 172.21.15.204:50024) 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faebed9d640 1 -- 172.21.15.204:0/3676285839 learned_addr learned my addr 172.21.15.204:0/3676285839 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 10 client.?.objecter ms_handle_connect 0x7faec009c8c0 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 10 client.?.objecter ms_handle_connect 0x7faec00b8240 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434945713 0 0) 0x7faec0133290 con 0x7faec009c8c0 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.159+0000 7faea77fe640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:34:32.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeac004eb0 con 0x7faec009c8c0 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 client.?.objecter ms_handle_connect 0x7faec0132da0 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201719389 0 0) 0x7faec01d7350 con 0x7faec00b8240 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec0133290 con 0x7faec00b8240 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211698634 0 0) 0x7faec0145090 con 0x7faec0132da0 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec01d7350 con 0x7faec0132da0 2024-03-29T11:34:32.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1787157992 0 0) 0x7faec0133290 con 0x7faec00b8240 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faec0145090 con 0x7faec00b8240 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faeb4001b80 con 0x7faec00b8240 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): dump: 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.172 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 851440427 0 0) 0x7faeac004eb0 con 0x7faec009c8c0 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faec0133290 con 0x7faec009c8c0 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faeb0001930 con 0x7faec009c8c0 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): dump: 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:32.173 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3427058255 0 0) 0x7faec01d7350 con 0x7faec0132da0 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faeac004eb0 con 0x7faec0132da0 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faeb8001bd0 con 0x7faec0132da0 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient(hunting): dump: 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.174 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3359850866 0 0) 0x7faec0145090 con 0x7faec00b8240 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 >> v1:172.21.15.22:6790/0 conn(0x7faec0132da0 legacy=0x7faec01d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 >> v1:172.21.15.204:6789/0 conn(0x7faec009c8c0 legacy=0x7faec0132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 monclient: found mon.a 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec0145220 con 0x7faec00b8240 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.175 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 5 monclient: authenticate success, global_id 4158 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 10 monclient: _renew_subs 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec0135420 con 0x7faec00b8240 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 10 monclient: _renew_subs 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faec01d7e20 con 0x7faec00b8240 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faec516b640 1 librados: init done 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faeb4004bf0 con 0x7faec00b8240 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faeb40010e0 con 0x7faec00b8240 2024-03-29T11:34:32.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faebd59a640 4 set_mon_vals no callback set 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 monclient: dump: 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faeb4005290 con 0x7faec00b8240 2024-03-29T11:34:32.177 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 10147922 0 0) 0x7faeb4006ad0 con 0x7faec00b8240 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 10 client.4158.objecter ms_dispatch 0x7faec0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 3 client.4158.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 3 client.4158.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.160+0000 7faea77fe640 20 client.4158.objecter dump_active .. 0 homeless 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.161+0000 7faec516b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.161+0000 7faec516b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.161+0000 7faec516b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.161+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea0005080 con 0x7faec00b8240 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.163+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faeb4006520 con 0x7faec00b8240 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.163+0000 7faea77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.178 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.163+0000 7faea77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:32.197 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.200+0000 7f062ea3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:34:32.219 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.222+0000 7f3548438040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:34:32.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.236+0000 7faec516b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f54e98c9-12b9-4c5b-8939-a54cafe12e72", "id": 7}] 2024-03-29T11:34:32.233 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.236+0000 7faec516b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f54e98c9-12b9-4c5b-8939-a54cafe12e72", "id": 7}] 2024-03-29T11:34:32.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.236+0000 7faec516b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.234 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.236+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f54e98c9-12b9-4c5b-8939-a54cafe12e72", "id": 7} v 0) v1 -- 0x7faea00058b0 con 0x7faec00b8240 2024-03-29T11:34:32.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faea77fe640 1 -- 172.21.15.204:0/3676285839 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f54e98c9-12b9-4c5b-8939-a54cafe12e72", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3454614060 0 1371665406) 0x7faeb4006790 con 0x7faec00b8240 2024-03-29T11:34:32.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faea77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f54e98c9-12b9-4c5b-8939-a54cafe12e72", "id": 7}] 2024-03-29T11:34:32.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faea77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:32.235 INFO:teuthology.orchestra.run.smithi204.stdout:7 2024-03-29T11:34:32.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 10 librados: watch_flush enter 2024-03-29T11:34:32.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 10 librados: watch_flush exit 2024-03-29T11:34:32.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 20 client.4158.objecter shutdown clearing up homeless session... 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 10 client.4158.objecter successfully canceled tick 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 10 monclient: shutdown 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 >> v1:172.21.15.22:6789/0 conn(0x7faec00b8240 legacy=0x7faec0134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 shutdown_connections 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 >> 172.21.15.204:0/3676285839 conn(0x7faec0066b00 msgr2=0x7faec013dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.238+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 shutdown_connections 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.239+0000 7faec516b640 1 -- 172.21.15.204:0/3676285839 wait complete. 2024-03-29T11:34:32.237 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-29T11:34:32.239+0000 7faec516b640 1 librados: shutdown 2024-03-29T11:34:32.246 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:34:32.246 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:34:32.248 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:34:32.248 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:34:32.249 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:34:32.250 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:34:32.251 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:34:32.251 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:34:32.252 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:34:32.253 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:34:32.253 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:34:32.255 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:34:32.255 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:34:32.255 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:34:32.288 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:34:32.289 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:34:32.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:34:32.291 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:34:32.291 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:34:32.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:34:32.293 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:34:32.293 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:34:32.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:34:32.295 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:34:32.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:32.338 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.341+0000 7f062ea3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:32.358 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.360+0000 7f3548438040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:34:32.389 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.393+0000 7f062ea3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:32.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.414+0000 7fca11b52640 10 monclient: get_monmap_and_config 2024-03-29T11:34:32.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.414+0000 7fca11b52640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.414+0000 7fca11b52640 10 monclient: monmap: 2024-03-29T11:34:32.411 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-29T11:34:32.411 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.411 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:34:32.414867+0000 2024-03-29T11:34:32.411 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:34:32.414867+0000 2024-03-29T11:34:32.411 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.412 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:32.412 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:32.412 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:32.412 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:32.412 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:32.418 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.416+0000 7f3548438040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:34:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.415+0000 7fca11b52640 1 Processor -- start 2024-03-29T11:34:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.415+0000 7fca11b52640 1 -- start start 2024-03-29T11:34:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.415+0000 7fca11b52640 10 monclient: init 2024-03-29T11:34:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.415+0000 7fca11b52640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.415+0000 7fca11b52640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.415+0000 7fca11b52640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c066920 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:32.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.415+0000 7fca11b52640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c066d00 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c1440b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0b0970 con 0x7fca0c1440b0 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c143480 con 0x7fca0c066d00 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c13ff80 con 0x7fca0c066920 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 10 monclient(hunting): authenticate will time out at 922.692070s 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fca0c066920 0x7fca0c13d550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34630/0 (socket says 172.21.15.22:34630) 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0b7fe640 1 -- 172.21.15.22:0/2805078353 learned_addr learned my addr 172.21.15.22:0/2805078353 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0bfff640 1 --1- 172.21.15.22:0/2805078353 >> v1:172.21.15.22:6789/0 conn(0x7fca0c1440b0 0x7fca0c14b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58240/0 (socket says 172.21.15.22:58240) 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443310593 0 0) 0x7fca0c13ff80 con 0x7fca0c066920 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f4003520 con 0x7fca0c066920 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578843064 0 0) 0x7fca0c0b0970 con 0x7fca0c1440b0 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c13ff80 con 0x7fca0c1440b0 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740274607 0 0) 0x7fca0c143480 con 0x7fca0c066d00 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c0b0970 con 0x7fca0c066d00 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1663372675 0 0) 0x7fc9f4003520 con 0x7fca0c066920 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 >> v1:172.21.15.204:6789/0 conn(0x7fca0c066d00 legacy=0x7fca0c1439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 >> v1:172.21.15.22:6789/0 conn(0x7fca0c1440b0 legacy=0x7fca0c14b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c140110 con 0x7fca0c066920 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc9fc001690 con 0x7fca0c066920 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: dump: 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.425 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9fc001870 con 0x7fca0c066920 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 1 -- 172.21.15.22:0/2805078353 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc9fc001b20 con 0x7fca0c066920 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca11b52640 4 set_mon_vals no callback set 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.416+0000 7fca0a7fc640 10 monclient: dump: 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.417+0000 7fca11b52640 10 monclient: shutdown 2024-03-29T11:34:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.417+0000 7fca11b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.417+0000 7fca11b52640 1 -- 172.21.15.22:0/2805078353 >> v1:172.21.15.22:6790/0 conn(0x7fca0c066920 legacy=0x7fca0c13d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.417+0000 7fca11b52640 1 -- 172.21.15.22:0/2805078353 shutdown_connections 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.417+0000 7fca11b52640 1 -- 172.21.15.22:0/2805078353 >> 172.21.15.22:0/2805078353 conn(0x7fca0c0b4760 msgr2=0x7fca0c0b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.417+0000 7fca11b52640 1 -- 172.21.15.22:0/2805078353 shutdown_connections 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.417+0000 7fca11b52640 1 -- 172.21.15.22:0/2805078353 wait complete. 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 10 monclient: build_initial_monmap 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 10 monclient: monmap: 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2024-03-29T11:34:32.428 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 1 librados: starting msgr at 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 1 librados: starting objecter 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 1 Processor -- start 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 1 -- start start 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 1 librados: setting wanted keys 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 1 librados: calling monclient init 2024-03-29T11:34:32.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.418+0000 7fca11b52640 10 monclient: init 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c066920 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c1440b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c0ae2d0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c142ef0 con 0x7fca0c066920 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c141280 con 0x7fca0c0ae2d0 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca0b7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fca0c066920 0x7fca0c1381b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58252/0 (socket says 172.21.15.22:58252) 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca0b7fe640 1 -- 172.21.15.22:0/2935177036 learned_addr learned my addr 172.21.15.22:0/2935177036 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c141760 con 0x7fca0c1440b0 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:32.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fca11b52640 10 monclient(hunting): authenticate will time out at 922.695430s 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 client.?.objecter ms_handle_connect 0x7fca0c066920 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 client.?.objecter ms_handle_connect 0x7fca0c1440b0 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 client.?.objecter ms_handle_connect 0x7fca0c0ae2d0 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973714498 0 0) 0x7fca0c142ef0 con 0x7fca0c066920 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f8004eb0 con 0x7fca0c066920 2024-03-29T11:34:32.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982996821 0 0) 0x7fca0c141760 con 0x7fca0c1440b0 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c142ef0 con 0x7fca0c1440b0 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824446037 0 0) 0x7fca0c141280 con 0x7fca0c0ae2d0 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c141760 con 0x7fca0c0ae2d0 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2001645155 0 0) 0x7fc9f8004eb0 con 0x7fca0c066920 2024-03-29T11:34:32.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c141280 con 0x7fca0c066920 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc9fc001710 con 0x7fca0c066920 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): dump: 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.433 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.419+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244336583 0 0) 0x7fca0c142ef0 con 0x7fca0c1440b0 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9f8004eb0 con 0x7fca0c1440b0 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 463635840 0 0) 0x7fca0c141760 con 0x7fca0c0ae2d0 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c142ef0 con 0x7fca0c0ae2d0 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fca00001870 con 0x7fca0c1440b0 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient(hunting): dump: 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:32.434 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fca040019b0 con 0x7fca0c0ae2d0 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:32.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient(hunting): dump: 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2205322878 0 0) 0x7fca0c141280 con 0x7fca0c066920 2024-03-29T11:34:32.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 >> v1:172.21.15.22:6790/0 conn(0x7fca0c1440b0 legacy=0x7fca0c06eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 >> v1:172.21.15.204:6789/0 conn(0x7fca0c0ae2d0 legacy=0x7fca0c1e6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 monclient: found mon.a 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c143160 con 0x7fca0c066920 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 10 monclient: _renew_subs 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c0a95e0 con 0x7fca0c066920 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9fc004a90 con 0x7fca0c066920 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc9fc004ea0 con 0x7fca0c066920 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 10 monclient: _renew_subs 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c138aa0 con 0x7fca0c066920 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca11b52640 1 librados: init done 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fca09ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:32.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.420+0000 7fc9f3fff640 10 monclient: dump: 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:32.439 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc9fc005390 con 0x7fca0c066920 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fc9fc006c80 con 0x7fca0c066920 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fc9f3fff640 10 client.4188.objecter ms_dispatch 0x7fca0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fc9f3fff640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fc9f3fff640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fc9f3fff640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fca11b52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fca11b52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fca11b52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.421+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c04f730 con 0x7fca0c066920 2024-03-29T11:34:32.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.423+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc9fc006570 con 0x7fca0c066920 2024-03-29T11:34:32.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.423+0000 7fc9f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:32.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.423+0000 7fc9f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:32.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.485+0000 7fca11b52640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:32.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.485+0000 7fca11b52640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:32.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.485+0000 7fca11b52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:34:32.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.485+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca0c06a420 con 0x7fca0c066920 2024-03-29T11:34:32.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fc9f3fff640 1 -- 172.21.15.22:0/2935177036 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fc9fc0068a0 con 0x7fca0c066920 2024-03-29T11:34:32.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fc9f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:32.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fc9f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:32.483 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T11:34:32.483 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:32.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 10 librados: watch_flush enter 2024-03-29T11:34:32.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 10 librados: watch_flush exit 2024-03-29T11:34:32.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T11:34:32.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 10 client.4188.objecter successfully canceled tick 2024-03-29T11:34:32.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 10 monclient: shutdown 2024-03-29T11:34:32.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:32.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 >> v1:172.21.15.22:6789/0 conn(0x7fca0c066920 legacy=0x7fca0c1381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:32.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 shutdown_connections 2024-03-29T11:34:32.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.486+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 >> 172.21.15.22:0/2935177036 conn(0x7fca0c0b4760 msgr2=0x7fca0c0b4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:32.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.487+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 shutdown_connections 2024-03-29T11:34:32.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.487+0000 7fca11b52640 1 -- 172.21.15.22:0/2935177036 wait complete. 2024-03-29T11:34:32.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T11:34:32.487+0000 7fca11b52640 1 librados: shutdown 2024-03-29T11:34:32.493 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:34:32.494 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:34:32.510 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.513+0000 7f062ea3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:32.526 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.529+0000 7f3548438040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:32.609 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.612+0000 7f062ea3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:34:32.623 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.626+0000 7f3548438040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:34:32.770 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.773+0000 7f062ea3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:32.780 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.783+0000 7f3548438040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:34:32.814 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.817+0000 7f062ea3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:32.823 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.826+0000 7f3548438040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:34:32.853 INFO:tasks.ceph.osd.0.smithi022.stderr:2024-03-29T11:34:32.856+0000 7f057d136640 -1 Falling back to public interface 2024-03-29T11:34:32.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2024-03-29T11:34:32.857+0000 7fdc8271f640 -1 Falling back to public interface 2024-03-29T11:34:32.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2024-03-29T11:34:32.868+0000 7f28510c8640 -1 Falling back to public interface 2024-03-29T11:34:32.871 INFO:tasks.ceph.osd.3.smithi022.stderr:2024-03-29T11:34:32.875+0000 7fa7f3437640 -1 Falling back to public interface 2024-03-29T11:34:32.884 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-03-29T11:34:32.887+0000 7ff814c1a640 -1 Falling back to public interface 2024-03-29T11:34:32.893 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-03-29T11:34:32.896+0000 7f6ce9614640 -1 Falling back to public interface 2024-03-29T11:34:32.895 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-03-29T11:34:32.898+0000 7f468d413640 -1 Falling back to public interface 2024-03-29T11:34:32.911 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-03-29T11:34:32.914+0000 7fc969c21640 -1 Falling back to public interface 2024-03-29T11:34:32.979 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:32.982+0000 7f062ea3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:32.982 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:32.985+0000 7f3548438040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:34:33.299 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:33.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.392+0000 7f1abacb3640 10 monclient: get_monmap_and_config 2024-03-29T11:34:33.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.392+0000 7f1abacb3640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.393+0000 7f1abacb3640 10 monclient: monmap: 2024-03-29T11:34:33.389 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:34:33.389 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:33.390 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:34:33.393258+0000 2024-03-29T11:34:33.390 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:34:33.393258+0000 2024-03-29T11:34:33.390 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.390 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:33.390 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:33.390 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:33.390 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:33.390 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:33.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.393+0000 7f1abacb3640 1 Processor -- start 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.393+0000 7f1abacb3640 1 -- start start 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.393+0000 7f1abacb3640 10 monclient: init 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 10 monclient(hunting): picked mon.noname-c con 0x7f1ab41457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 10 monclient(hunting): picked mon.noname-b con 0x7f1ab413f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 10 monclient(hunting): picked mon.noname-a con 0x7f1ab40a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab40aaef0 con 0x7f1ab40a9d70 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab4145bd0 con 0x7f1ab41457f0 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab4145eb0 con 0x7f1ab413f650 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1abacb3640 10 monclient(hunting): authenticate will time out at 923.670412s 2024-03-29T11:34:33.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab3fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f1ab413f650 0x7f1ab40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56306/0 (socket says 172.21.15.22:56306) 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab9250640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f1ab40a9d70 0x7f1ab414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38130/0 (socket says 172.21.15.22:38130) 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab3fff640 1 -- 172.21.15.22:0/3734620167 learned_addr learned my addr 172.21.15.22:0/3734620167 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932578591 0 0) 0x7f1ab40aaef0 con 0x7f1ab40a9d70 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa4003520 con 0x7f1ab40a9d70 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080043483 0 0) 0x7f1ab4145eb0 con 0x7f1ab413f650 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:34:33.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab40aaef0 con 0x7f1ab413f650 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567200270 0 0) 0x7f1ab4145bd0 con 0x7f1ab41457f0 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.394+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab4145eb0 con 0x7f1ab41457f0 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20229196 0 0) 0x7f1ab40aaef0 con 0x7f1ab413f650 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 >> v1:172.21.15.204:6789/0 conn(0x7f1ab41457f0 legacy=0x7f1ab413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 >> v1:172.21.15.22:6789/0 conn(0x7f1ab40a9d70 legacy=0x7f1ab414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 1 monclient: found mon.noname-b 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4146340 con 0x7f1ab413f650 2024-03-29T11:34:33.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1aa0001670 con 0x7f1ab413f650 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: dump: 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:33.403 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1aa00017d0 con 0x7f1ab413f650 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 1 -- 172.21.15.22:0/3734620167 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1aa0001c20 con 0x7f1ab413f650 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 4 set_mon_vals no callback set 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1ab37fe640 10 monclient: dump: 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:33.404 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 10 monclient: shutdown 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 1 -- 172.21.15.22:0/3734620167 >> v1:172.21.15.22:6790/0 conn(0x7f1ab413f650 legacy=0x7f1ab40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 1 -- 172.21.15.22:0/3734620167 shutdown_connections 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 1 -- 172.21.15.22:0/3734620167 >> 172.21.15.22:0/3734620167 conn(0x7f1ab4066a00 msgr2=0x7f1ab4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 1 -- 172.21.15.22:0/3734620167 shutdown_connections 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.395+0000 7f1abacb3640 1 -- 172.21.15.22:0/3734620167 wait complete. 2024-03-29T11:34:33.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 10 monclient: build_initial_monmap 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 10 monclient: monmap: 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 1 librados: starting msgr at 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 1 librados: starting objecter 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 1 Processor -- start 2024-03-29T11:34:33.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 1 -- start start 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 1 librados: setting wanted keys 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 1 librados: calling monclient init 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.396+0000 7f1abacb3640 10 monclient: init 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 10 monclient(hunting): picked mon.noname-a con 0x7f1ab40a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 10 monclient(hunting): picked mon.noname-c con 0x7f1ab413f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 10 monclient(hunting): picked mon.noname-b con 0x7f1ab41457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab4141c60 con 0x7f1ab40a9d70 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab413fda0 con 0x7f1ab41457f0 2024-03-29T11:34:33.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab4140080 con 0x7f1ab413f650 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1abacb3640 10 monclient(hunting): authenticate will time out at 923.673654s 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab3fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f1ab413f650 0x7f1ab41422d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56308/0 (socket says 172.21.15.22:56308) 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab3fff640 1 -- 172.21.15.22:0/3371848089 learned_addr learned my addr 172.21.15.22:0/3371848089 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 client.?.objecter ms_handle_connect 0x7f1ab413f650 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 client.?.objecter ms_handle_connect 0x7f1ab40a9d70 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 client.?.objecter ms_handle_connect 0x7f1ab41457f0 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503489293 0 0) 0x7f1ab4140080 con 0x7f1ab413f650 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.397+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a98004eb0 con 0x7f1ab413f650 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288646521 0 0) 0x7f1ab4141c60 con 0x7f1ab40a9d70 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab4140080 con 0x7f1ab40a9d70 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292029349 0 0) 0x7f1ab413fda0 con 0x7f1ab41457f0 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:34:33.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab4141c60 con 0x7f1ab41457f0 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160733044 0 0) 0x7f1a98004eb0 con 0x7f1ab413f650 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ab413fda0 con 0x7f1ab413f650 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644169895 0 0) 0x7f1ab4140080 con 0x7f1ab40a9d70 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a98004eb0 con 0x7f1ab40a9d70 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1aa0001630 con 0x7f1ab413f650 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): dump: 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.410 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 495041030 0 0) 0x7f1ab4141c60 con 0x7f1ab41457f0 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ab4140080 con 0x7f1ab41457f0 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1aa80017d0 con 0x7f1ab40a9d70 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): dump: 2024-03-29T11:34:33.411 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1aac0015b0 con 0x7f1ab41457f0 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient(hunting): dump: 2024-03-29T11:34:33.412 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2246052582 0 0) 0x7f1ab413fda0 con 0x7f1ab413f650 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 >> v1:172.21.15.204:6789/0 conn(0x7f1ab41457f0 legacy=0x7f1ab41e1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 >> v1:172.21.15.22:6789/0 conn(0x7f1ab40a9d70 legacy=0x7f1ab4140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 monclient: found mon.c 2024-03-29T11:34:33.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4141eb0 con 0x7f1ab413f650 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 10 monclient: _renew_subs 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab4140c90 con 0x7f1ab413f650 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 10 monclient: _renew_subs 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab4142bc0 con 0x7f1ab413f650 2024-03-29T11:34:33.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1abacb3640 1 librados: init done 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1aa00011f0 con 0x7f1ab413f650 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1aa0004cf0 con 0x7f1ab413f650 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab2ffd640 4 set_mon_vals no callback set 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 monclient: dump: 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:33.415 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1aa00052a0 con 0x7f1ab413f650 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f1aa00064d0 con 0x7f1ab413f650 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 10 client.4190.objecter ms_dispatch 0x7f1ab4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.398+0000 7f1ab17fa640 20 client.4190.objecter dump_active .. 0 homeless 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.399+0000 7f1abacb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.399+0000 7f1abacb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.399+0000 7f1abacb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.399+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a94005080 con 0x7f1ab413f650 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.403+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1aa00066e0 con 0x7f1ab413f650 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.403+0000 7f1ab17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:33.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.403+0000 7f1ab17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.465+0000 7f1abacb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.465+0000 7f1abacb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.465+0000 7f1abacb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.465+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a94005330 con 0x7f1ab413f650 2024-03-29T11:34:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.466+0000 7f1ab17fa640 1 -- 172.21.15.22:0/3371848089 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f1aa0006960 con 0x7f1ab413f650 2024-03-29T11:34:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.466+0000 7f1ab17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.466+0000 7f1ab17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:33.463 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:34:33.463 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 10 librados: watch_flush enter 2024-03-29T11:34:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 10 librados: watch_flush exit 2024-03-29T11:34:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 20 client.4190.objecter shutdown clearing up homeless session... 2024-03-29T11:34:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 10 client.4190.objecter successfully canceled tick 2024-03-29T11:34:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 10 monclient: shutdown 2024-03-29T11:34:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 >> v1:172.21.15.22:6790/0 conn(0x7f1ab413f650 legacy=0x7f1ab41422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 shutdown_connections 2024-03-29T11:34:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 >> 172.21.15.22:0/3371848089 conn(0x7f1ab4066a00 msgr2=0x7f1ab40b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 shutdown_connections 2024-03-29T11:34:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 1 -- 172.21.15.22:0/3371848089 wait complete. 2024-03-29T11:34:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:33.467+0000 7f1abacb3640 1 librados: shutdown 2024-03-29T11:34:33.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:33.811 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:33.813+0000 7f3548438040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:33.823 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:33.826+0000 7f062ea3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:34:34.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:34.092+0000 7f3548438040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:34.111 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:34.114+0000 7f062ea3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:34:34.141 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:34.144+0000 7f3548438040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:34.164 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:34.167+0000 7f062ea3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:34:34.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:34.259+0000 7f3548438040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:34.287 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T11:34:34.291+0000 7f062ea3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:34:35.271 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:35.274+0000 7f34e903d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:34:35.272 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-29T11:34:35.274+0000 7f34e903d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:34:36.314 INFO:tasks.ceph.osd.0.smithi022.stderr:2024-03-29T11:34:36.317+0000 7f057d136640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:34:36.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2024-03-29T11:34:36.367+0000 7fdc8271f640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:34:36.381 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-03-29T11:34:36.384+0000 7f6ce9614640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:34:36.382 INFO:tasks.ceph.osd.3.smithi022.stderr:2024-03-29T11:34:36.386+0000 7fa7f3437640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:34:36.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2024-03-29T11:34:36.396+0000 7f28510c8640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:34:36.446 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-03-29T11:34:36.449+0000 7ff814c1a640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:34:36.448 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-03-29T11:34:36.451+0000 7f468d413640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:34:36.449 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-03-29T11:34:36.452+0000 7fc969c21640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:34:40.279 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.431+0000 7f236433b640 10 monclient: get_monmap_and_config 2024-03-29T11:34:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.431+0000 7f236433b640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.431+0000 7f236433b640 10 monclient: monmap: 2024-03-29T11:34:40.428 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:34:40.431923+0000 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:34:40.431923+0000 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:40.429 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:40.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.432+0000 7f236433b640 1 Processor -- start 2024-03-29T11:34:40.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.432+0000 7f236433b640 1 -- start start 2024-03-29T11:34:40.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.432+0000 7f236433b640 10 monclient: init 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.433+0000 7f236433b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.433+0000 7f236433b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.433+0000 7f236433b640 10 monclient(hunting): picked mon.noname-b con 0x7f235c06d9d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f236433b640 10 monclient(hunting): picked mon.noname-a con 0x7f235c13ef30 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f236433b640 10 monclient(hunting): picked mon.noname-c con 0x7f235c0b07d0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f236433b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235c0aed10 con 0x7f235c13ef30 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f236433b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235c0b0c40 con 0x7f235c0b07d0 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f236433b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235c0ab1b0 con 0x7f235c06d9d0 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f236433b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f236433b640 10 monclient(hunting): authenticate will time out at 930.710204s 2024-03-29T11:34:40.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f23628d8640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f235c0b07d0 0x7f235c14bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:42194/0 (socket says 172.21.15.22:42194) 2024-03-29T11:34:40.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.434+0000 7f23628d8640 1 -- 172.21.15.22:0/2119880521 learned_addr learned my addr 172.21.15.22:0/2119880521 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:40.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462244090 0 0) 0x7f235c0b0c40 con 0x7f235c0b07d0 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2348003520 con 0x7f235c0b07d0 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 605825509 0 0) 0x7f2348003520 con 0x7f235c0b07d0 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 >> v1:172.21.15.22:6790/0 conn(0x7f235c06d9d0 legacy=0x7f235c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 >> v1:172.21.15.22:6789/0 conn(0x7f235c13ef30 legacy=0x7f235c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 monclient: found mon.noname-c 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c0ab490 con 0x7f235c0b07d0 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2354001a10 con 0x7f235c0b07d0 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: dump: 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:40.434 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f236433b640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f236433b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:40.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2354001c30 con 0x7f235c0b07d0 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 1 -- 172.21.15.22:0/2119880521 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2354001fc0 con 0x7f235c0b07d0 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.435+0000 7f23610d5640 10 monclient: dump: 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:40.436 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:40.437 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:40.437 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:40.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.436+0000 7f236433b640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:40.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.436+0000 7f236433b640 4 set_mon_vals no callback set 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.436+0000 7f236433b640 10 monclient: shutdown 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.436+0000 7f236433b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.436+0000 7f236433b640 1 -- 172.21.15.22:0/2119880521 >> v1:172.21.15.204:6789/0 conn(0x7f235c0b07d0 legacy=0x7f235c14bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.437+0000 7f236433b640 1 -- 172.21.15.22:0/2119880521 shutdown_connections 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.437+0000 7f236433b640 1 -- 172.21.15.22:0/2119880521 >> 172.21.15.22:0/2119880521 conn(0x7f235c06b660 msgr2=0x7f235c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.437+0000 7f236433b640 1 -- 172.21.15.22:0/2119880521 shutdown_connections 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.437+0000 7f236433b640 1 -- 172.21.15.22:0/2119880521 wait complete. 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.438+0000 7f236433b640 10 monclient: build_initial_monmap 2024-03-29T11:34:40.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.438+0000 7f236433b640 10 monclient: monmap: 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.438+0000 7f236433b640 1 librados: starting msgr at 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.438+0000 7f236433b640 1 librados: starting objecter 2024-03-29T11:34:40.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.439+0000 7f236433b640 1 Processor -- start 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.439+0000 7f236433b640 1 -- start start 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.439+0000 7f236433b640 1 librados: setting wanted keys 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.439+0000 7f236433b640 1 librados: calling monclient init 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.439+0000 7f236433b640 10 monclient: init 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.440+0000 7f236433b640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.440+0000 7f236433b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.440+0000 7f236433b640 10 monclient(hunting): picked mon.noname-a con 0x7f235c06d9d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.440+0000 7f236433b640 10 monclient(hunting): picked mon.noname-c con 0x7f235c0b07d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.440+0000 7f23620d7640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f235c06d9d0 0x7f235c06e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38822/0 (socket says 172.21.15.22:38822) 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f236433b640 10 monclient(hunting): picked mon.noname-b con 0x7f235c13ef30 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f23620d7640 1 -- 172.21.15.22:0/1661124475 learned_addr learned my addr 172.21.15.22:0/1661124475 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235c06e730 con 0x7f235c06d9d0 2024-03-29T11:34:40.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235c1dbbc0 con 0x7f235c13ef30 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 client.?.objecter ms_handle_connect 0x7f235c06d9d0 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 client.?.objecter ms_handle_connect 0x7f235c13ef30 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235c0b43b0 con 0x7f235c0b07d0 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f236433b640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f236433b640 10 monclient(hunting): authenticate will time out at 930.717721s 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 client.?.objecter ms_handle_connect 0x7f235c0b07d0 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596704855 0 0) 0x7f235c1dbbc0 con 0x7f235c13ef30 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.441+0000 7f2352ffd640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2340004eb0 con 0x7f235c13ef30 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2488867322 0 0) 0x7f2340004eb0 con 0x7f235c13ef30 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f235c1dbbc0 con 0x7f235c13ef30 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f23540018d0 con 0x7f235c13ef30 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 10 monclient(hunting): dump: 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:40.450 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684950316 0 0) 0x7f235c0ab1b0 con 0x7f235c06d9d0 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.442+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2340004eb0 con 0x7f235c06d9d0 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895918466 0 0) 0x7f235c0b43b0 con 0x7f235c0b07d0 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f235c0ab1b0 con 0x7f235c0b07d0 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863458208 0 0) 0x7f235c1dbbc0 con 0x7f235c13ef30 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 >> v1:172.21.15.22:6790/0 conn(0x7f235c0b07d0 legacy=0x7f235c1dc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 >> v1:172.21.15.22:6789/0 conn(0x7f235c06d9d0 legacy=0x7f235c06e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:40.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 1 monclient: found mon.b 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.443+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c1dc760 con 0x7f235c13ef30 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f2352ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f2352ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f2352ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 10 monclient: _renew_subs 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235c06edd0 con 0x7f235c13ef30 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:40.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 10 monclient: _renew_subs 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f235c1d87c0 con 0x7f235c13ef30 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f236433b640 1 librados: init done 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2354001a30 con 0x7f235c13ef30 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f2352ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.444+0000 7f23608d4640 4 set_mon_vals no callback set 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.445+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2354005110 con 0x7f235c13ef30 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.445+0000 7f2352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.445+0000 7f2352ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.445+0000 7f2352ffd640 10 monclient: dump: 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:40.454 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.445+0000 7f236433b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.446+0000 7f236433b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.446+0000 7f236433b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.446+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2344005080 con 0x7f235c13ef30 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.447+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f235401b0e0 con 0x7f235c13ef30 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.448+0000 7f23620d7640 10 monclient: get_auth_request con 0x7f234006dee0 auth_method 0 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.448+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f2354084e70 con 0x7f235c13ef30 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.448+0000 7f2352ffd640 10 client.4237.objecter ms_dispatch 0x7f235c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.448+0000 7f2352ffd640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.448+0000 7f2352ffd640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.448+0000 7f2352ffd640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T11:34:40.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.448+0000 7f2352ffd640 10 client.4237.objecter ms_handle_connect 0x7f234006dee0 2024-03-29T11:34:40.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.454+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2354050750 con 0x7f235c13ef30 2024-03-29T11:34:40.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.454+0000 7f2352ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:40.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.454+0000 7f2352ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:40.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.882+0000 7f236433b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:40.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.882+0000 7f236433b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:40.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.882+0000 7f236433b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:34:40.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.882+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2344005330 con 0x7f235c13ef30 2024-03-29T11:34:40.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.883+0000 7f2352ffd640 1 -- 172.21.15.22:0/1661124475 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f2354052680 con 0x7f235c13ef30 2024-03-29T11:34:40.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.883+0000 7f2352ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:40.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.883+0000 7f2352ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:40.880 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:34:40.880 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 10 librados: watch_flush enter 2024-03-29T11:34:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 10 librados: watch_flush exit 2024-03-29T11:34:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T11:34:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 10 client.4237.objecter successfully canceled tick 2024-03-29T11:34:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 >> v1:172.21.15.204:6816/3848099522 conn(0x7f234006dee0 legacy=0x7f23400703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 10 monclient: shutdown 2024-03-29T11:34:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 >> v1:172.21.15.204:6789/0 conn(0x7f235c13ef30 legacy=0x7f235c0b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 shutdown_connections 2024-03-29T11:34:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 >> 172.21.15.22:0/1661124475 conn(0x7f235c06b660 msgr2=0x7f235c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 shutdown_connections 2024-03-29T11:34:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 1 -- 172.21.15.22:0/1661124475 wait complete. 2024-03-29T11:34:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:40.887+0000 7f236433b640 1 librados: shutdown 2024-03-29T11:34:40.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:47.707 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:47.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.862+0000 7fc21e64d640 10 monclient: get_monmap_and_config 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.862+0000 7fc21e64d640 10 monclient: build_initial_monmap 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.862+0000 7fc21e64d640 10 monclient: monmap: 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:34:47.863975+0000 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:34:47.863975+0000 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:47.867 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.863+0000 7fc21e64d640 1 Processor -- start 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.863+0000 7fc21e64d640 1 -- start start 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.863+0000 7fc21e64d640 10 monclient: init 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.864+0000 7fc21e64d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.864+0000 7fc21e64d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.864+0000 7fc21e64d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2180b0860 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc21e64d640 10 monclient(hunting): picked mon.noname-c con 0x7fc21813ef30 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc21e64d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2180af8e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc217fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc2180b0860 0x7fc21813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45376/0 (socket says 172.21.15.22:45376) 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc2177fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc21813ef30 0x7fc2180af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:47254/0 (socket says 172.21.15.22:47254) 2024-03-29T11:34:47.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc217fff640 1 -- 172.21.15.22:0/1718413452 learned_addr learned my addr 172.21.15.22:0/1718413452 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc21e64d640 1 -- 172.21.15.22:0/1718413452 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2180b0c40 con 0x7fc2180af8e0 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc21e64d640 1 -- 172.21.15.22:0/1718413452 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2180aed10 con 0x7fc21813ef30 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc21e64d640 1 -- 172.21.15.22:0/1718413452 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2180afd50 con 0x7fc2180b0860 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc21e64d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc21e64d640 10 monclient(hunting): authenticate will time out at 938.142569s 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090475700 0 0) 0x7fc2180b0c40 con 0x7fc2180af8e0 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.865+0000 7fc216ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc200003440 con 0x7fc2180af8e0 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081047501 0 0) 0x7fc2180afd50 con 0x7fc2180b0860 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:47.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2180b0c40 con 0x7fc2180b0860 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3448953889 0 0) 0x7fc200003440 con 0x7fc2180af8e0 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 >> v1:172.21.15.22:6790/0 conn(0x7fc2180b0860 legacy=0x7fc21813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 >> v1:172.21.15.204:6789/0 conn(0x7fc21813ef30 legacy=0x7fc2180af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 monclient: found mon.noname-a 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2180ab1b0 con 0x7fc2180af8e0 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:47.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc208001610 con 0x7fc2180af8e0 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: dump: 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:47.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc21e64d640 5 monclient: authenticate success, global_id 4242 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc21e64d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc208001770 con 0x7fc2180af8e0 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 1 -- 172.21.15.22:0/1718413452 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc208001bc0 con 0x7fc2180af8e0 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.866+0000 7fc216ffd640 10 monclient: dump: 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:47.872 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.867+0000 7fc21e64d640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.867+0000 7fc21e64d640 4 set_mon_vals no callback set 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.867+0000 7fc21e64d640 10 monclient: shutdown 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.867+0000 7fc21e64d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.867+0000 7fc21e64d640 1 -- 172.21.15.22:0/1718413452 >> v1:172.21.15.22:6789/0 conn(0x7fc2180af8e0 legacy=0x7fc218149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.868+0000 7fc21e64d640 1 -- 172.21.15.22:0/1718413452 shutdown_connections 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.868+0000 7fc21e64d640 1 -- 172.21.15.22:0/1718413452 >> 172.21.15.22:0/1718413452 conn(0x7fc21806b660 msgr2=0x7fc21806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.868+0000 7fc21e64d640 1 -- 172.21.15.22:0/1718413452 shutdown_connections 2024-03-29T11:34:47.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.868+0000 7fc21e64d640 1 -- 172.21.15.22:0/1718413452 wait complete. 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.869+0000 7fc21e64d640 10 monclient: build_initial_monmap 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.869+0000 7fc21e64d640 10 monclient: monmap: 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:47.874 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:47.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.869+0000 7fc21e64d640 1 librados: starting msgr at 2024-03-29T11:34:47.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.869+0000 7fc21e64d640 1 librados: starting objecter 2024-03-29T11:34:47.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.869+0000 7fc21e64d640 1 Processor -- start 2024-03-29T11:34:47.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.870+0000 7fc21e64d640 1 -- start start 2024-03-29T11:34:47.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.870+0000 7fc21e64d640 1 librados: setting wanted keys 2024-03-29T11:34:47.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.870+0000 7fc21e64d640 1 librados: calling monclient init 2024-03-29T11:34:47.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.870+0000 7fc21e64d640 10 monclient: init 2024-03-29T11:34:47.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.870+0000 7fc21e64d640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 10 monclient(hunting): picked mon.noname-c con 0x7fc2180af8e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2180b0860 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 10 monclient(hunting): picked mon.noname-a con 0x7fc21813ef30 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21cbea640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fc21813ef30 0x7fc2181c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34058/0 (socket says 172.21.15.22:34058) 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21cbea640 1 -- 172.21.15.22:0/1136249356 learned_addr learned my addr 172.21.15.22:0/1136249356 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc214ff9640 10 client.?.objecter ms_handle_connect 0x7fc21813ef30 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc214ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2181d76e0 con 0x7fc21813ef30 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2181d7340 con 0x7fc2180b0860 2024-03-29T11:34:47.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc218134e40 con 0x7fc2180af8e0 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.871+0000 7fc21e64d640 10 monclient(hunting): authenticate will time out at 938.148888s 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 client.?.objecter ms_handle_connect 0x7fc2180af8e0 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707509360 0 0) 0x7fc2181d76e0 con 0x7fc21813ef30 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc204004eb0 con 0x7fc21813ef30 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948117281 0 0) 0x7fc218134e40 con 0x7fc2180af8e0 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:47.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2181d76e0 con 0x7fc2180af8e0 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387291466 0 0) 0x7fc2181d76e0 con 0x7fc2180af8e0 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc218134e40 con 0x7fc2180af8e0 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc210001cb0 con 0x7fc2180af8e0 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:47.885 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 client.?.objecter ms_handle_connect 0x7fc2180b0860 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4142201455 0 0) 0x7fc204004eb0 con 0x7fc21813ef30 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2181d76e0 con 0x7fc21813ef30 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc208001b80 con 0x7fc21813ef30 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:47.886 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095301899 0 0) 0x7fc2181d7340 con 0x7fc2180b0860 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:34:47.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc204004eb0 con 0x7fc2180b0860 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 442409145 0 0) 0x7fc218134e40 con 0x7fc2180af8e0 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 >> v1:172.21.15.204:6789/0 conn(0x7fc2180b0860 legacy=0x7fc218134730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 >> v1:172.21.15.22:6789/0 conn(0x7fc21813ef30 legacy=0x7fc2181c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 1 monclient: found mon.c 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.872+0000 7fc214ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc218134fd0 con 0x7fc2180af8e0 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 5 monclient: authenticate success, global_id 4238 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 10 monclient: _renew_subs 2024-03-29T11:34:47.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2181351c0 con 0x7fc2180af8e0 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 10 monclient: _renew_subs 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc218139b60 con 0x7fc2180af8e0 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc21e64d640 1 librados: init done 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc210004b60 con 0x7fc2180af8e0 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc210004e70 con 0x7fc2180af8e0 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc214ff9640 10 monclient: dump: 2024-03-29T11:34:47.889 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.873+0000 7fc2167fc640 4 set_mon_vals no callback set 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fc21001ae40 con 0x7fc2180af8e0 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc21e64d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:47.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc21e64d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc21e64d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f8005080 con 0x7fc2180af8e0 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fc210084d00 con 0x7fc2180af8e0 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc214ff9640 10 client.4238.objecter ms_dispatch 0x7fc218000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc214ff9640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc214ff9640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.875+0000 7fc214ff9640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.876+0000 7fc2177fe640 10 monclient: get_auth_request con 0x7fc2040767f0 auth_method 0 2024-03-29T11:34:47.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.876+0000 7fc214ff9640 10 client.4238.objecter ms_handle_connect 0x7fc2040767f0 2024-03-29T11:34:47.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.888+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2100505e0 con 0x7fc2180af8e0 2024-03-29T11:34:47.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.888+0000 7fc214ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:47.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:47.888+0000 7fc214ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:48.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.264+0000 7fc21e64d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:48.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.264+0000 7fc21e64d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:48.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.264+0000 7fc21e64d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:48.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.264+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1f8005330 con 0x7fc2180af8e0 2024-03-29T11:34:48.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.265+0000 7fc214ff9640 1 -- 172.21.15.22:0/1136249356 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fc210052510 con 0x7fc2180af8e0 2024-03-29T11:34:48.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.265+0000 7fc214ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:48.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.265+0000 7fc214ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:48.263 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:34:48.263 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:48.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.268+0000 7fc21e64d640 10 librados: watch_flush enter 2024-03-29T11:34:48.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.268+0000 7fc21e64d640 10 librados: watch_flush exit 2024-03-29T11:34:48.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.268+0000 7fc21e64d640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T11:34:48.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.268+0000 7fc21e64d640 10 client.4238.objecter successfully canceled tick 2024-03-29T11:34:48.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.269+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 >> v1:172.21.15.204:6816/3848099522 conn(0x7fc2040767f0 legacy=0x7fc204078cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:48.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.269+0000 7fc21e64d640 10 monclient: shutdown 2024-03-29T11:34:48.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.269+0000 7fc21e64d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:48.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.269+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 >> v1:172.21.15.22:6790/0 conn(0x7fc2180af8e0 legacy=0x7fc2181d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:48.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.269+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 shutdown_connections 2024-03-29T11:34:48.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.269+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 >> 172.21.15.22:0/1136249356 conn(0x7fc21806b660 msgr2=0x7fc21806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:48.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.270+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 shutdown_connections 2024-03-29T11:34:48.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.270+0000 7fc21e64d640 1 -- 172.21.15.22:0/1136249356 wait complete. 2024-03-29T11:34:48.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:48.271+0000 7fc21e64d640 1 librados: shutdown 2024-03-29T11:34:48.285 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:55.092 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:55.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.240+0000 7fde9f5e2640 10 monclient: get_monmap_and_config 2024-03-29T11:34:55.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.240+0000 7fde9f5e2640 10 monclient: build_initial_monmap 2024-03-29T11:34:55.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.240+0000 7fde9f5e2640 10 monclient: monmap: 2024-03-29T11:34:55.242 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:34:55.242 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:55.242 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:34:55.242050+0000 2024-03-29T11:34:55.242 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:34:55.242050+0000 2024-03-29T11:34:55.242 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.242+0000 7fde9f5e2640 1 Processor -- start 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.242+0000 7fde9f5e2640 1 -- start start 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.242+0000 7fde9f5e2640 10 monclient: init 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 10 monclient(hunting): picked mon.noname-c con 0x7fde980ad6c0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 10 monclient(hunting): picked mon.noname-a con 0x7fde980af6c0 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 10 monclient(hunting): picked mon.noname-b con 0x7fde9814fdc0 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:55.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde980a94e0 con 0x7fde980af6c0 2024-03-29T11:34:55.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde980af3a0 con 0x7fde980ad6c0 2024-03-29T11:34:55.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde980afb60 con 0x7fde9814fdc0 2024-03-29T11:34:55.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:55.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.243+0000 7fde9f5e2640 10 monclient(hunting): authenticate will time out at 945.520573s 2024-03-29T11:34:55.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.245+0000 7fde9d37e640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fde980ad6c0 0x7fde9814f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:34362/0 (socket says 172.21.15.22:34362) 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.245+0000 7fde9d37e640 1 -- 172.21.15.22:0/163421554 learned_addr learned my addr 172.21.15.22:0/163421554 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546942407 0 0) 0x7fde980af3a0 con 0x7fde980ad6c0 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde84003520 con 0x7fde980ad6c0 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218317658 0 0) 0x7fde980afb60 con 0x7fde9814fdc0 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:34:55.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.246+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde980af3a0 con 0x7fde9814fdc0 2024-03-29T11:34:55.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.249+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666887717 0 0) 0x7fde980a94e0 con 0x7fde980af6c0 2024-03-29T11:34:55.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.249+0000 7fde8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:55.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.249+0000 7fde8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:55.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.249+0000 7fde8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:55.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.249+0000 7fde8ffff640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.249+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde980afb60 con 0x7fde980af6c0 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.249+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190138770 0 0) 0x7fde980afb60 con 0x7fde980af6c0 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 >> v1:172.21.15.22:6790/0 conn(0x7fde9814fdc0 legacy=0x7fde9815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 >> v1:172.21.15.204:6789/0 conn(0x7fde980ad6c0 legacy=0x7fde9814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 1 monclient: found mon.noname-a 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde980aab80 con 0x7fde980af6c0 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: _finish_auth 0 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:55.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde9f5e2640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde9f5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fde88001570 con 0x7fde980af6c0 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: dump: 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:55.251 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde9f5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde880016d0 con 0x7fde980af6c0 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 1 -- 172.21.15.22:0/163421554 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fde88001b20 con 0x7fde980af6c0 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde8ffff640 10 monclient: dump: 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:55.252 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde9f5e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.250+0000 7fde9f5e2640 4 set_mon_vals no callback set 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.251+0000 7fde9f5e2640 10 monclient: shutdown 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.251+0000 7fde9f5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.251+0000 7fde9f5e2640 1 -- 172.21.15.22:0/163421554 >> v1:172.21.15.22:6789/0 conn(0x7fde980af6c0 legacy=0x7fde98152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.251+0000 7fde9f5e2640 1 -- 172.21.15.22:0/163421554 shutdown_connections 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.251+0000 7fde9f5e2640 1 -- 172.21.15.22:0/163421554 >> 172.21.15.22:0/163421554 conn(0x7fde980b35e0 msgr2=0x7fde980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.251+0000 7fde9f5e2640 1 -- 172.21.15.22:0/163421554 shutdown_connections 2024-03-29T11:34:55.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.252+0000 7fde9f5e2640 1 -- 172.21.15.22:0/163421554 wait complete. 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.252+0000 7fde9f5e2640 10 monclient: build_initial_monmap 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.252+0000 7fde9f5e2640 10 monclient: monmap: 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:34:55.257 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.253+0000 7fde9f5e2640 1 librados: starting msgr at 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.253+0000 7fde9f5e2640 1 librados: starting objecter 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.253+0000 7fde9f5e2640 1 Processor -- start 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.253+0000 7fde9f5e2640 1 -- start start 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.253+0000 7fde9f5e2640 1 librados: setting wanted keys 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.253+0000 7fde9f5e2640 1 librados: calling monclient init 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.253+0000 7fde9f5e2640 10 monclient: init 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 10 monclient(hunting): picked mon.noname-a con 0x7fde980ad6c0 addr v1:172.21.15.22:6789/0 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 10 monclient(hunting): picked mon.noname-b con 0x7fde9814fdc0 addr v1:172.21.15.204:6789/0 2024-03-29T11:34:55.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 10 monclient(hunting): picked mon.noname-c con 0x7fde9814e890 addr v1:172.21.15.22:6790/0 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde980703c0 con 0x7fde980ad6c0 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9808a990 con 0x7fde9814fdc0 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9808a640 con 0x7fde9814e890 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.254+0000 7fde9f5e2640 10 monclient(hunting): authenticate will time out at 945.531383s 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde9db7f640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fde9814e890 0x7fde98200590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35705/0 (socket says 172.21.15.22:35705) 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde9db7f640 1 -- 172.21.15.22:0/875593085 learned_addr learned my addr 172.21.15.22:0/875593085 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde8dffb640 10 client.?.objecter ms_handle_connect 0x7fde9814e890 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461269252 0 0) 0x7fde9808a640 con 0x7fde9814e890 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:55.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.256+0000 7fde8dffb640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde7c004eb0 con 0x7fde9814e890 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3410657403 0 0) 0x7fde7c004eb0 con 0x7fde9814e890 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde9808a640 con 0x7fde9814e890 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fde90001c10 con 0x7fde9814e890 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 10 monclient(hunting): dump: 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:55.260 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3517461698 0 0) 0x7fde9808a640 con 0x7fde9814e890 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 >> v1:172.21.15.204:6789/0 conn(0x7fde9814fdc0 legacy=0x7fde98071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 >> v1:172.21.15.22:6789/0 conn(0x7fde980ad6c0 legacy=0x7fde981f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 1 monclient: found mon.c 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.257+0000 7fde8dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde981f53c0 con 0x7fde9814e890 2024-03-29T11:34:55.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde8dffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde8dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 10 monclient: _renew_subs 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde981f5670 con 0x7fde9814e890 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 10 monclient: _renew_subs 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde98072410 con 0x7fde9814e890 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde9f5e2640 1 librados: init done 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde90001de0 con 0x7fde9814e890 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.258+0000 7fde8effd640 4 set_mon_vals no callback set 2024-03-29T11:34:55.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.259+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fde90003c20 con 0x7fde9814e890 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.259+0000 7fde8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.259+0000 7fde8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.259+0000 7fde8dffb640 10 monclient: dump: 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:34:55.263 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.261+0000 7fde9f5e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.261+0000 7fde9f5e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.261+0000 7fde9f5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.261+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde78005080 con 0x7fde9814e890 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.262+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fde90003d80 con 0x7fde9814e890 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.262+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fde90084dd0 con 0x7fde9814e890 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.262+0000 7fde8dffb640 10 client.4244.objecter ms_dispatch 0x7fde98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.262+0000 7fde8dffb640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.262+0000 7fde8dffb640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:55.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.263+0000 7fde8dffb640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T11:34:55.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.263+0000 7fde9d37e640 10 monclient: get_auth_request con 0x7fde7c06dda0 auth_method 0 2024-03-29T11:34:55.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.263+0000 7fde8dffb640 10 client.4244.objecter ms_handle_connect 0x7fde7c06dda0 2024-03-29T11:34:55.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.271+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde900504e0 con 0x7fde9814e890 2024-03-29T11:34:55.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.271+0000 7fde8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:55.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.271+0000 7fde8dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:55.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.591+0000 7fde9f5e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:55.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.591+0000 7fde9f5e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:55.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.591+0000 7fde9f5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:34:55.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.591+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde78005330 con 0x7fde9814e890 2024-03-29T11:34:55.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.592+0000 7fde8dffb640 1 -- 172.21.15.22:0/875593085 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fde900525e0 con 0x7fde9814e890 2024-03-29T11:34:55.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.592+0000 7fde8dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:55.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.592+0000 7fde8dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:55.592 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:34:55.592 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:55.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.597+0000 7fde9f5e2640 10 librados: watch_flush enter 2024-03-29T11:34:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.599+0000 7fde9f5e2640 10 librados: watch_flush exit 2024-03-29T11:34:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.599+0000 7fde9f5e2640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T11:34:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.600+0000 7fde9f5e2640 10 client.4244.objecter successfully canceled tick 2024-03-29T11:34:55.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.600+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 >> v1:172.21.15.204:6816/3848099522 conn(0x7fde7c06dda0 legacy=0x7fde7c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.600+0000 7fde9f5e2640 10 monclient: shutdown 2024-03-29T11:34:55.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.600+0000 7fde9f5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:55.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.600+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 >> v1:172.21.15.22:6790/0 conn(0x7fde9814e890 legacy=0x7fde98200590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:55.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.600+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 shutdown_connections 2024-03-29T11:34:55.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.600+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 >> 172.21.15.22:0/875593085 conn(0x7fde980b35e0 msgr2=0x7fde980ac1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:55.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.601+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 shutdown_connections 2024-03-29T11:34:55.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.601+0000 7fde9f5e2640 1 -- 172.21.15.22:0/875593085 wait complete. 2024-03-29T11:34:55.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:34:55.601+0000 7fde9f5e2640 1 librados: shutdown 2024-03-29T11:34:55.618 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:02.425 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:02.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.575+0000 7f60ccc4a640 10 monclient: get_monmap_and_config 2024-03-29T11:35:02.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.575+0000 7f60ccc4a640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.575+0000 7f60ccc4a640 10 monclient: monmap: 2024-03-29T11:35:02.573 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:02.573 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:02.573 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:35:02.576746+0000 2024-03-29T11:35:02.574 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:35:02.576746+0000 2024-03-29T11:35:02.574 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.574 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:02.574 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:02.574 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:35:02.574 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:35:02.574 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:02.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.576+0000 7f60ccc4a640 1 Processor -- start 2024-03-29T11:35:02.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.576+0000 7f60ccc4a640 1 -- start start 2024-03-29T11:35:02.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.576+0000 7f60ccc4a640 10 monclient: init 2024-03-29T11:35:02.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.578+0000 7f60ccc4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.578+0000 7f60ccc4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:02.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.578+0000 7f60ccc4a640 10 monclient(hunting): picked mon.noname-a con 0x7f60c81457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:02.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.579+0000 7f60ccc4a640 10 monclient(hunting): picked mon.noname-c con 0x7f60c813f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:02.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.579+0000 7f60ccc4a640 10 monclient(hunting): picked mon.noname-b con 0x7f60c80a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:02.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.581+0000 7f60ccc4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c80aaef0 con 0x7f60c81457f0 2024-03-29T11:35:02.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.581+0000 7f60ccc4a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c8145bd0 con 0x7f60c813f650 2024-03-29T11:35:02.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.581+0000 7f60ccc4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c8145eb0 con 0x7f60c80a9d70 2024-03-29T11:35:02.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.581+0000 7f60ccc4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.581+0000 7f60ccc4a640 10 monclient(hunting): authenticate will time out at 952.858305s 2024-03-29T11:35:02.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.581+0000 7f60c5d9b640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f60c813f650 0x7f60c80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:42370/0 (socket says 172.21.15.22:42370) 2024-03-29T11:35:02.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.581+0000 7f60c5d9b640 1 -- 172.21.15.22:0/2752082994 learned_addr learned my addr 172.21.15.22:0/2752082994 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:02.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.583+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395294492 0 0) 0x7f60c8145bd0 con 0x7f60c813f650 2024-03-29T11:35:02.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.583+0000 7f60c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.583+0000 7f60c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.583+0000 7f60c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.583+0000 7f60c559a640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:35:02.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.583+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b0003520 con 0x7f60c813f650 2024-03-29T11:35:02.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510886009 0 0) 0x7f60c8145eb0 con 0x7f60c80a9d70 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c8145bd0 con 0x7f60c80a9d70 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404647820 0 0) 0x7f60c80aaef0 con 0x7f60c81457f0 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c8145eb0 con 0x7f60c81457f0 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3698870187 0 0) 0x7f60c8145bd0 con 0x7f60c80a9d70 2024-03-29T11:35:02.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 >> v1:172.21.15.204:6789/0 conn(0x7f60c813f650 legacy=0x7f60c80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 >> v1:172.21.15.22:6789/0 conn(0x7f60c81457f0 legacy=0x7f60c813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 monclient: found mon.noname-b 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8146340 con 0x7f60c80a9d70 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.585+0000 7f60c559a640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60ccc4a640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60ccc4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f60c0001920 con 0x7f60c80a9d70 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 10 monclient: dump: 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60c0001b40 con 0x7f60c80a9d70 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 1 -- 172.21.15.22:0/2752082994 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f60c0001ed0 con 0x7f60c80a9d70 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60c559a640 10 monclient: dump: 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:02.588 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:02.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60ccc4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:02.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.586+0000 7f60ccc4a640 4 set_mon_vals no callback set 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.587+0000 7f60ccc4a640 10 monclient: shutdown 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.587+0000 7f60ccc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.587+0000 7f60ccc4a640 1 -- 172.21.15.22:0/2752082994 >> v1:172.21.15.22:6790/0 conn(0x7f60c80a9d70 legacy=0x7f60c814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.587+0000 7f60ccc4a640 1 -- 172.21.15.22:0/2752082994 shutdown_connections 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.587+0000 7f60ccc4a640 1 -- 172.21.15.22:0/2752082994 >> 172.21.15.22:0/2752082994 conn(0x7f60c8066a00 msgr2=0x7f60c8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.587+0000 7f60ccc4a640 1 -- 172.21.15.22:0/2752082994 shutdown_connections 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.587+0000 7f60ccc4a640 1 -- 172.21.15.22:0/2752082994 wait complete. 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.589+0000 7f60ccc4a640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.589+0000 7f60ccc4a640 10 monclient: monmap: 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.589+0000 7f60ccc4a640 1 librados: starting msgr at 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.589+0000 7f60ccc4a640 1 librados: starting objecter 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.589+0000 7f60ccc4a640 1 Processor -- start 2024-03-29T11:35:02.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.590+0000 7f60ccc4a640 1 -- start start 2024-03-29T11:35:02.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.590+0000 7f60ccc4a640 1 librados: setting wanted keys 2024-03-29T11:35:02.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.590+0000 7f60ccc4a640 1 librados: calling monclient init 2024-03-29T11:35:02.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.590+0000 7f60ccc4a640 10 monclient: init 2024-03-29T11:35:02.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.590+0000 7f60ccc4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.590+0000 7f60ccc4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:02.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.591+0000 7f60ccc4a640 10 monclient(hunting): picked mon.noname-c con 0x7f60c80a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:02.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60ccc4a640 10 monclient(hunting): picked mon.noname-b con 0x7f60c813f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:02.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60ccc4a640 10 monclient(hunting): picked mon.noname-a con 0x7f60c81457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:02.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60ccc4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c8142490 con 0x7f60c81457f0 2024-03-29T11:35:02.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60ccc4a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c8137890 con 0x7f60c813f650 2024-03-29T11:35:02.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60ccc4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c8141690 con 0x7f60c80a9d70 2024-03-29T11:35:02.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60ccc4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60ccc4a640 10 monclient(hunting): authenticate will time out at 952.870228s 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60c5d9b640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f60c813f650 0x7f60c8136a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:43816/0 (socket says 172.21.15.22:43816) 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.593+0000 7f60c5d9b640 1 -- 172.21.15.22:0/262048260 learned_addr learned my addr 172.21.15.22:0/262048260 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.594+0000 7f60b6ffd640 10 client.?.objecter ms_handle_connect 0x7f60c813f650 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.594+0000 7f60b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.595+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733541655 0 0) 0x7f60c8137890 con 0x7f60c813f650 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.595+0000 7f60b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.595+0000 7f60b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.595+0000 7f60b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.595+0000 7f60b6ffd640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:35:02.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.595+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a8004eb0 con 0x7f60c813f650 2024-03-29T11:35:02.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.596+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2343074004 0 0) 0x7f60a8004eb0 con 0x7f60c813f650 2024-03-29T11:35:02.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.596+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60c8137890 con 0x7f60c813f650 2024-03-29T11:35:02.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.596+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f60b8001950 con 0x7f60c813f650 2024-03-29T11:35:02.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.596+0000 7f60b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.596+0000 7f60b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.596+0000 7f60b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.596+0000 7f60b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 10 client.?.objecter ms_handle_connect 0x7f60c81457f0 2024-03-29T11:35:02.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1398671220 0 0) 0x7f60c8137890 con 0x7f60c813f650 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 >> v1:172.21.15.22:6790/0 conn(0x7f60c80a9d70 legacy=0x7f60c813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 >> v1:172.21.15.22:6789/0 conn(0x7f60c81457f0 legacy=0x7f60c8137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 1 monclient: found mon.b 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8142a80 con 0x7f60c813f650 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.597+0000 7f60b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60b80047e0 con 0x7f60c813f650 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60c4d99640 4 set_mon_vals no callback set 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f60b8001950 con 0x7f60c813f650 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60b6ffd640 10 monclient: dump: 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:02.600 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 10 monclient: _renew_subs 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c8141e20 con 0x7f60c813f650 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:02.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 10 monclient: _renew_subs 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c8144740 con 0x7f60c813f650 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.598+0000 7f60ccc4a640 1 librados: init done 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.600+0000 7f60ccc4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.600+0000 7f60ccc4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.600+0000 7f60ccc4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.600+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c8142640 con 0x7f60c813f650 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.601+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f60b8004a90 con 0x7f60c813f650 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.601+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f60b8084940 con 0x7f60c813f650 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.602+0000 7f60b6ffd640 10 client.4249.objecter ms_dispatch 0x7f60c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.602+0000 7f60b6ffd640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.602+0000 7f60b6ffd640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.602+0000 7f60b6ffd640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-29T11:35:02.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.602+0000 7f60c659c640 10 monclient: get_auth_request con 0x7f60a806ddc0 auth_method 0 2024-03-29T11:35:02.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.604+0000 7f60b6ffd640 10 client.4249.objecter ms_handle_connect 0x7f60a806ddc0 2024-03-29T11:35:02.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.610+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60c8142640 con 0x7f60c813f650 2024-03-29T11:35:02.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.610+0000 7f60b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.610+0000 7f60b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:02.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.948+0000 7f60ccc4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.948+0000 7f60ccc4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.948+0000 7f60ccc4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:02.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.948+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60c80ac0f0 con 0x7f60c813f650 2024-03-29T11:35:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.950+0000 7f60b6ffd640 1 -- 172.21.15.22:0/262048260 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f60c8142640 con 0x7f60c813f650 2024-03-29T11:35:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.950+0000 7f60b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.950+0000 7f60b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:02.948 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:35:02.948 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.953+0000 7f60ccc4a640 10 librados: watch_flush enter 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 10 librados: watch_flush exit 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 10 client.4249.objecter successfully canceled tick 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 >> v1:172.21.15.204:6816/3848099522 conn(0x7f60a806ddc0 legacy=0x7f60a8070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 10 monclient: shutdown 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 >> v1:172.21.15.204:6789/0 conn(0x7f60c813f650 legacy=0x7f60c8136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 shutdown_connections 2024-03-29T11:35:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 >> 172.21.15.22:0/262048260 conn(0x7f60c8066a00 msgr2=0x7f60c8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 shutdown_connections 2024-03-29T11:35:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 1 -- 172.21.15.22:0/262048260 wait complete. 2024-03-29T11:35:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:02.955+0000 7f60ccc4a640 1 librados: shutdown 2024-03-29T11:35:02.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:09.776 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:09.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.927+0000 7f2aa2616640 10 monclient: get_monmap_and_config 2024-03-29T11:35:09.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.927+0000 7f2aa2616640 10 monclient: build_initial_monmap 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.927+0000 7f2aa2616640 10 monclient: monmap: 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:35:09.928591+0000 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:35:09.928591+0000 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:35:09.926 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:09.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.928+0000 7f2aa2616640 1 Processor -- start 2024-03-29T11:35:09.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.928+0000 7f2aa2616640 1 -- start start 2024-03-29T11:35:09.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.928+0000 7f2aa2616640 10 monclient: init 2024-03-29T11:35:09.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.929+0000 7f2aa2616640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:09.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.929+0000 7f2aa2616640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:09.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.929+0000 7f2aa2616640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9c0b0860 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:09.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.932+0000 7f2aa2616640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9c13ef30 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:09.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.932+0000 7f2aa2616640 10 monclient(hunting): picked mon.noname-b con 0x7f2a9c0af8e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:09.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.934+0000 7f2aa2616640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0b0c40 con 0x7f2a9c13ef30 2024-03-29T11:35:09.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.934+0000 7f2aa2616640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0aed10 con 0x7f2a9c0b0860 2024-03-29T11:35:09.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.934+0000 7f2aa2616640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0afd50 con 0x7f2a9c0af8e0 2024-03-29T11:35:09.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.934+0000 7f2aa2616640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:09.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.934+0000 7f2aa2616640 10 monclient(hunting): authenticate will time out at 960.211366s 2024-03-29T11:35:09.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.934+0000 7f2a9b7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f2a9c13ef30 0x7f2a9c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:52887/0 (socket says 172.21.15.22:52887) 2024-03-29T11:35:09.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.934+0000 7f2a9b7fe640 1 -- 172.21.15.22:0/2491756452 learned_addr learned my addr 172.21.15.22:0/2491756452 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869074047 0 0) 0x7f2a9c0b0c40 con 0x7f2a9c13ef30 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a84003520 con 0x7f2a9c13ef30 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971361037 0 0) 0x7f2a9c0aed10 con 0x7f2a9c0b0860 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.935+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a9c0b0c40 con 0x7f2a9c0b0860 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 623820239 0 0) 0x7f2a84003520 con 0x7f2a9c13ef30 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 >> v1:172.21.15.22:6790/0 conn(0x7f2a9c0af8e0 legacy=0x7f2a9c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 >> v1:172.21.15.204:6789/0 conn(0x7f2a9c0b0860 legacy=0x7f2a9c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 1 monclient: found mon.noname-a 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0ab1b0 con 0x7f2a9c13ef30 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2aa2616640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:35:09.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2aa2616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2a90001690 con 0x7f2a9c13ef30 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2a9affd640 10 monclient: dump: 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:09.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.936+0000 7f2aa2616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:09.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.939+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a900017f0 con 0x7f2a9c13ef30 2024-03-29T11:35:09.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.939+0000 7f2a9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:09.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.939+0000 7f2aa2616640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.939+0000 7f2aa2616640 4 set_mon_vals no callback set 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.939+0000 7f2a9affd640 1 -- 172.21.15.22:0/2491756452 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2a90001c00 con 0x7f2a9c13ef30 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.939+0000 7f2a9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.940+0000 7f2a9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.940+0000 7f2a9affd640 10 monclient: dump: 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:09.941 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:09.942 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:09.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.940+0000 7f2aa2616640 10 monclient: shutdown 2024-03-29T11:35:09.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.940+0000 7f2aa2616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:09.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.941+0000 7f2aa2616640 1 -- 172.21.15.22:0/2491756452 >> v1:172.21.15.22:6789/0 conn(0x7f2a9c13ef30 legacy=0x7f2a9c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.941+0000 7f2aa2616640 1 -- 172.21.15.22:0/2491756452 shutdown_connections 2024-03-29T11:35:09.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.941+0000 7f2aa2616640 1 -- 172.21.15.22:0/2491756452 >> 172.21.15.22:0/2491756452 conn(0x7f2a9c06b660 msgr2=0x7f2a9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:09.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.941+0000 7f2aa2616640 1 -- 172.21.15.22:0/2491756452 shutdown_connections 2024-03-29T11:35:09.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.941+0000 7f2aa2616640 1 -- 172.21.15.22:0/2491756452 wait complete. 2024-03-29T11:35:09.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.944+0000 7f2aa2616640 10 monclient: build_initial_monmap 2024-03-29T11:35:09.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.944+0000 7f2aa2616640 10 monclient: monmap: 2024-03-29T11:35:09.943 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:09.943 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:09.943 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.944+0000 7f2aa2616640 1 librados: starting msgr at 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.944+0000 7f2aa2616640 1 librados: starting objecter 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.945+0000 7f2aa2616640 1 Processor -- start 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.945+0000 7f2aa2616640 1 -- start start 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.945+0000 7f2aa2616640 1 librados: setting wanted keys 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.945+0000 7f2aa2616640 1 librados: calling monclient init 2024-03-29T11:35:09.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.945+0000 7f2aa2616640 10 monclient: init 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.946+0000 7f2aa2616640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9c0af8e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 10 monclient(hunting): picked mon.noname-b con 0x7f2a9c0b0860 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9c13ef30 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0b2980 con 0x7f2a9c0af8e0 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0b4250 con 0x7f2a9c0b0860 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9c0b43e0 con 0x7f2a9c13ef30 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:09.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.947+0000 7f2aa2616640 10 monclient(hunting): authenticate will time out at 960.224398s 2024-03-29T11:35:09.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.952+0000 7f2a9b7fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f2a9c0b0860 0x7f2a9c136770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:58641/0 (socket says 172.21.15.22:58641) 2024-03-29T11:35:09.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.952+0000 7f2a9b7fe640 1 -- 172.21.15.22:0/2649570067 learned_addr learned my addr 172.21.15.22:0/2649570067 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:09.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.954+0000 7f2a98ff9640 10 client.?.objecter ms_handle_connect 0x7f2a9c0b0860 2024-03-29T11:35:09.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.954+0000 7f2a98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:09.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.954+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705226909 0 0) 0x7f2a9c0b4250 con 0x7f2a9c0b0860 2024-03-29T11:35:09.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.954+0000 7f2a98ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.954+0000 7f2a98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.954+0000 7f2a98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.954+0000 7f2a98ff9640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.954+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a88004eb0 con 0x7f2a9c0b0860 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3538499496 0 0) 0x7f2a88004eb0 con 0x7f2a9c0b0860 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a9c0b4250 con 0x7f2a9c0b0860 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2a90001bc0 con 0x7f2a9c0b0860 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:09.954 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4159315388 0 0) 0x7f2a9c0b4250 con 0x7f2a9c0b0860 2024-03-29T11:35:09.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.955+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 >> v1:172.21.15.22:6790/0 conn(0x7f2a9c13ef30 legacy=0x7f2a9c136e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 >> v1:172.21.15.22:6789/0 conn(0x7f2a9c0af8e0 legacy=0x7f2a9c0b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 1 monclient: found mon.b 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c0b15f0 con 0x7f2a9c0b0860 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2a98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:09.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2aa2616640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2aa2616640 10 monclient: _renew_subs 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.956+0000 7f2aa2616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9c0b1780 con 0x7f2a9c0b0860 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2aa2616640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2aa2616640 10 monclient: _renew_subs 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2aa2616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c1dff00 con 0x7f2a9c0b0860 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2aa2616640 1 librados: init done 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a90004b00 con 0x7f2a9c0b0860 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2a98ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2a90004f10 con 0x7f2a9c0b0860 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2a98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.958+0000 7f2a98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:09.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.957+0000 7f2a9a7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.959+0000 7f2aa2616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.958+0000 7f2a98ff9640 10 monclient: dump: 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:09.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.959+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f2a90005070 con 0x7f2a9c0b0860 2024-03-29T11:35:09.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.961+0000 7f2aa2616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:09.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.961+0000 7f2aa2616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:09.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.961+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f2a90084c00 con 0x7f2a9c0b0860 2024-03-29T11:35:09.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.961+0000 7f2a98ff9640 10 client.4255.objecter ms_dispatch 0x7f2a9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:09.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.961+0000 7f2a98ff9640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:09.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.961+0000 7f2a98ff9640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:09.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.961+0000 7f2a98ff9640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T11:35:09.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.961+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c005080 con 0x7f2a9c0b0860 2024-03-29T11:35:09.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.964+0000 7f2a9bfff640 10 monclient: get_auth_request con 0x7f2a8806dc70 auth_method 0 2024-03-29T11:35:09.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.965+0000 7f2a98ff9640 10 client.4255.objecter ms_handle_connect 0x7f2a8806dc70 2024-03-29T11:35:09.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.971+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a900504e0 con 0x7f2a9c0b0860 2024-03-29T11:35:09.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.971+0000 7f2a98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:09.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:09.971+0000 7f2a98ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:10.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.297+0000 7f2aa2616640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.297+0000 7f2aa2616640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.297+0000 7f2aa2616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:10.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.297+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a7c005330 con 0x7f2a9c0b0860 2024-03-29T11:35:10.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.300+0000 7f2a98ff9640 1 -- 172.21.15.22:0/2649570067 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f2a90052410 con 0x7f2a9c0b0860 2024-03-29T11:35:10.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.300+0000 7f2a98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.300+0000 7f2a98ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:10.299 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:35:10.299 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:10.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.304+0000 7f2aa2616640 10 librados: watch_flush enter 2024-03-29T11:35:10.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.304+0000 7f2aa2616640 10 librados: watch_flush exit 2024-03-29T11:35:10.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.304+0000 7f2aa2616640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T11:35:10.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.304+0000 7f2aa2616640 10 client.4255.objecter successfully canceled tick 2024-03-29T11:35:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.306+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 >> v1:172.21.15.204:6816/3848099522 conn(0x7f2a8806dc70 legacy=0x7f2a88070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.306+0000 7f2aa2616640 10 monclient: shutdown 2024-03-29T11:35:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.306+0000 7f2aa2616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.306+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 >> v1:172.21.15.204:6789/0 conn(0x7f2a9c0b0860 legacy=0x7f2a9c136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.306+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 shutdown_connections 2024-03-29T11:35:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.306+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 >> 172.21.15.22:0/2649570067 conn(0x7f2a9c06b660 msgr2=0x7f2a9c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.307+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 shutdown_connections 2024-03-29T11:35:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.307+0000 7f2aa2616640 1 -- 172.21.15.22:0/2649570067 wait complete. 2024-03-29T11:35:10.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:10.309+0000 7f2aa2616640 1 librados: shutdown 2024-03-29T11:35:10.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:17.130 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.286+0000 7fc1db91c640 10 monclient: get_monmap_and_config 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.286+0000 7fc1db91c640 10 monclient: build_initial_monmap 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.286+0000 7fc1db91c640 10 monclient: monmap: 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:35:17.287436+0000 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:35:17.287436+0000 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:17.285 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:17.286 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:35:17.286 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:35:17.286 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:17.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.287+0000 7fc1db91c640 1 Processor -- start 2024-03-29T11:35:17.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 1 -- start start 2024-03-29T11:35:17.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 10 monclient: init 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 10 monclient(hunting): picked mon.noname-a con 0x7fc1d40ae250 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 10 monclient(hunting): picked mon.noname-c con 0x7fc1d413f970 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 10 monclient(hunting): picked mon.noname-b con 0x7fc1d4142800 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d40afe00 con 0x7fc1d40ae250 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d4142c10 con 0x7fc1d413f970 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d40afb20 con 0x7fc1d4142800 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:17.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.288+0000 7fc1db91c640 10 monclient(hunting): authenticate will time out at 967.565961s 2024-03-29T11:35:17.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.295+0000 7fc1d8eb7640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc1d413f970 0x7fc1d41420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:35257/0 (socket says 172.21.15.22:35257) 2024-03-29T11:35:17.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.295+0000 7fc1d8eb7640 1 -- 172.21.15.22:0/1086847985 learned_addr learned my addr 172.21.15.22:0/1086847985 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832206660 0 0) 0x7fc1d40afe00 con 0x7fc1d40ae250 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c0003520 con 0x7fc1d40ae250 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802139574 0 0) 0x7fc1d4142c10 con 0x7fc1d413f970 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d40afe00 con 0x7fc1d413f970 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049419041 0 0) 0x7fc1d40afb20 con 0x7fc1d4142800 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.297+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d4142c10 con 0x7fc1d4142800 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1844635175 0 0) 0x7fc1c0003520 con 0x7fc1d40ae250 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 >> v1:172.21.15.22:6790/0 conn(0x7fc1d4142800 legacy=0x7fc1d414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 >> v1:172.21.15.204:6789/0 conn(0x7fc1d413f970 legacy=0x7fc1d41420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 1 monclient: found mon.noname-a 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:17.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40aab80 con 0x7fc1d40ae250 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1db91c640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1db91c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc1c4001690 con 0x7fc1d40ae250 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 10 monclient: dump: 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:17.303 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1db91c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 1 -- 172.21.15.22:0/1086847985 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1c40017f0 con 0x7fc1d40ae250 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1db91c640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.298+0000 7fc1db91c640 4 set_mon_vals no callback set 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.299+0000 7fc1db91c640 10 monclient: shutdown 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.299+0000 7fc1db91c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:17.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.299+0000 7fc1db91c640 1 -- 172.21.15.22:0/1086847985 >> v1:172.21.15.22:6789/0 conn(0x7fc1d40ae250 legacy=0x7fc1d413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.300+0000 7fc1db91c640 1 -- 172.21.15.22:0/1086847985 shutdown_connections 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.300+0000 7fc1db91c640 1 -- 172.21.15.22:0/1086847985 >> 172.21.15.22:0/1086847985 conn(0x7fc1d40b35e0 msgr2=0x7fc1d40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.300+0000 7fc1db91c640 1 -- 172.21.15.22:0/1086847985 shutdown_connections 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.300+0000 7fc1db91c640 1 -- 172.21.15.22:0/1086847985 wait complete. 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 10 monclient: build_initial_monmap 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 10 monclient: monmap: 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:35:17.305 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 1 librados: starting msgr at 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 1 librados: starting objecter 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 1 Processor -- start 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 1 -- start start 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 1 librados: setting wanted keys 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 1 librados: calling monclient init 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.301+0000 7fc1db91c640 10 monclient: init 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 10 monclient(hunting): picked mon.noname-a con 0x7fc1d40ae250 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 10 monclient(hunting): picked mon.noname-c con 0x7fc1d413f970 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 10 monclient(hunting): picked mon.noname-b con 0x7fc1d4142800 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:17.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d413c2b0 con 0x7fc1d40ae250 2024-03-29T11:35:17.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d413be90 con 0x7fc1d4142800 2024-03-29T11:35:17.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d4139870 con 0x7fc1d413f970 2024-03-29T11:35:17.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:17.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.302+0000 7fc1db91c640 10 monclient(hunting): authenticate will time out at 967.579689s 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.307+0000 7fc1d9eb9640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc1d4142800 0x7fc1d41f54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:35337/0 (socket says 172.21.15.22:35337) 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.307+0000 7fc1d9eb9640 1 -- 172.21.15.22:0/1274983353 learned_addr learned my addr 172.21.15.22:0/1274983353 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.307+0000 7fc1c9ffb640 10 client.?.objecter ms_handle_connect 0x7fc1d4142800 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.307+0000 7fc1c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793675904 0 0) 0x7fc1d413be90 con 0x7fc1d4142800 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b8004eb0 con 0x7fc1d4142800 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3469566698 0 0) 0x7fc1b8004eb0 con 0x7fc1d4142800 2024-03-29T11:35:17.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1d413be90 con 0x7fc1d4142800 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc1d00018c0 con 0x7fc1d4142800 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:17.310 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.308+0000 7fc1c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 10 client.?.objecter ms_handle_connect 0x7fc1d40ae250 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484740730 0 0) 0x7fc1d413c2b0 con 0x7fc1d40ae250 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b8004eb0 con 0x7fc1d40ae250 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.309+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2097290089 0 0) 0x7fc1b8004eb0 con 0x7fc1d40ae250 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.310+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1d413c2b0 con 0x7fc1d40ae250 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.310+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc1c4001670 con 0x7fc1d40ae250 2024-03-29T11:35:17.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.310+0000 7fc1c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.310+0000 7fc1c9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.310+0000 7fc1c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.310+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1120635424 0 0) 0x7fc1d413be90 con 0x7fc1d4142800 2024-03-29T11:35:17.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.310+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 >> v1:172.21.15.22:6790/0 conn(0x7fc1d413f970 legacy=0x7fc1d41f4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:17.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.311+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 >> v1:172.21.15.22:6789/0 conn(0x7fc1d40ae250 legacy=0x7fc1d41346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.311+0000 7fc1c9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:17.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.311+0000 7fc1c9ffb640 1 monclient: found mon.b 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.311+0000 7fc1c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.311+0000 7fc1c9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.311+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d4139a00 con 0x7fc1d4142800 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.311+0000 7fc1c9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1c9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 10 monclient: _renew_subs 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d4139c50 con 0x7fc1d4142800 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 10 monclient: _renew_subs 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:17.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d413a3a0 con 0x7fc1d4142800 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.312+0000 7fc1db91c640 1 librados: init done 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.313+0000 7fc1db91c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.313+0000 7fc1db91c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.313+0000 7fc1db91c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.313+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d4075f10 con 0x7fc1d4142800 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.314+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1d0001a20 con 0x7fc1d4142800 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.314+0000 7fc1c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.314+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc1d00038f0 con 0x7fc1d4142800 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.314+0000 7fc1c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.314+0000 7fc1c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.314+0000 7fc1c9ffb640 10 monclient: dump: 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:17.317 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.314+0000 7fc1cb7fe640 4 set_mon_vals no callback set 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.315+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fc1d0003b10 con 0x7fc1d4142800 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.315+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fc1d0084a00 con 0x7fc1d4142800 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.315+0000 7fc1c9ffb640 10 client.4261.objecter ms_dispatch 0x7fc1d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.316+0000 7fc1c9ffb640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.316+0000 7fc1c9ffb640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.316+0000 7fc1c9ffb640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:35:17.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.316+0000 7fc1d96b8640 10 monclient: get_auth_request con 0x7fc1b806dfa0 auth_method 0 2024-03-29T11:35:17.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.317+0000 7fc1c9ffb640 10 client.4261.objecter ms_handle_connect 0x7fc1b806dfa0 2024-03-29T11:35:17.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.322+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1d00502e0 con 0x7fc1d4142800 2024-03-29T11:35:17.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.322+0000 7fc1c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.322+0000 7fc1c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:17.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.647+0000 7fc1db91c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:17.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.647+0000 7fc1db91c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:17.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.647+0000 7fc1db91c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:17.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.647+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1d413c650 con 0x7fc1d4142800 2024-03-29T11:35:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.649+0000 7fc1c9ffb640 1 -- 172.21.15.22:0/1274983353 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fc1d0052210 con 0x7fc1d4142800 2024-03-29T11:35:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.649+0000 7fc1c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.649+0000 7fc1c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:17.649 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:35:17.649 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:17.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.654+0000 7fc1db91c640 10 librados: watch_flush enter 2024-03-29T11:35:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.654+0000 7fc1db91c640 10 librados: watch_flush exit 2024-03-29T11:35:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.654+0000 7fc1db91c640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:35:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.654+0000 7fc1db91c640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:35:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.654+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 >> v1:172.21.15.204:6816/3848099522 conn(0x7fc1b806dfa0 legacy=0x7fc1b8070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.655+0000 7fc1db91c640 10 monclient: shutdown 2024-03-29T11:35:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.655+0000 7fc1db91c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.655+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 >> v1:172.21.15.204:6789/0 conn(0x7fc1d4142800 legacy=0x7fc1d41f54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.657+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 shutdown_connections 2024-03-29T11:35:17.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.657+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 >> 172.21.15.22:0/1274983353 conn(0x7fc1d40b35e0 msgr2=0x7fc1d40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:17.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.657+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 shutdown_connections 2024-03-29T11:35:17.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.657+0000 7fc1db91c640 1 -- 172.21.15.22:0/1274983353 wait complete. 2024-03-29T11:35:17.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:17.657+0000 7fc1db91c640 1 librados: shutdown 2024-03-29T11:35:17.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:24.477 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:24.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.631+0000 7fe722ec0640 10 monclient: get_monmap_and_config 2024-03-29T11:35:24.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.631+0000 7fe722ec0640 10 monclient: build_initial_monmap 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.631+0000 7fe722ec0640 10 monclient: monmap: 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:35:24.632824+0000 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:35:24.632824+0000 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:35:24.630 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:24.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.632+0000 7fe722ec0640 1 Processor -- start 2024-03-29T11:35:24.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.633+0000 7fe722ec0640 1 -- start start 2024-03-29T11:35:24.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.633+0000 7fe722ec0640 10 monclient: init 2024-03-29T11:35:24.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.633+0000 7fe722ec0640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:24.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.633+0000 7fe722ec0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:24.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.633+0000 7fe722ec0640 10 monclient(hunting): picked mon.noname-c con 0x7fe71c1422b0 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:24.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.633+0000 7fe722ec0640 10 monclient(hunting): picked mon.noname-b con 0x7fe71c142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:24.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.634+0000 7fe722ec0640 10 monclient(hunting): picked mon.noname-a con 0x7fe71c0b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:24.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.634+0000 7fe722ec0640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c142c80 con 0x7fe71c0b17d0 2024-03-29T11:35:24.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.634+0000 7fe722ec0640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c0aae40 con 0x7fe71c1422b0 2024-03-29T11:35:24.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.634+0000 7fe722ec0640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c13fbd0 con 0x7fe71c142690 2024-03-29T11:35:24.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.634+0000 7fe722ec0640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:24.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.634+0000 7fe722ec0640 10 monclient(hunting): authenticate will time out at 974.911465s 2024-03-29T11:35:24.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.637+0000 7fe72145d640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fe71c0b17d0 0x7fe71c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:47489/0 (socket says 172.21.15.22:47489) 2024-03-29T11:35:24.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.637+0000 7fe72145d640 1 -- 172.21.15.22:0/2959170747 learned_addr learned my addr 172.21.15.22:0/2959170747 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.638+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868151416 0 0) 0x7fe71c142c80 con 0x7fe71c0b17d0 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.638+0000 7fe71b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.638+0000 7fe71b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.638+0000 7fe71b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.638+0000 7fe71b7fe640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.638+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe704003520 con 0x7fe71c0b17d0 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.639+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029631129 0 0) 0x7fe71c0aae40 con 0x7fe71c1422b0 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.639+0000 7fe71b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:24.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.639+0000 7fe71b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:24.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.639+0000 7fe71b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:24.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.639+0000 7fe71b7fe640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:35:24.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.639+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c142c80 con 0x7fe71c1422b0 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 319387833 0 0) 0x7fe704003520 con 0x7fe71c0b17d0 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 >> v1:172.21.15.22:6790/0 conn(0x7fe71c142690 legacy=0x7fe71c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 >> v1:172.21.15.204:6789/0 conn(0x7fe71c1422b0 legacy=0x7fe71c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c151940 con 0x7fe71c0b17d0 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:24.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe714001a10 con 0x7fe71c0b17d0 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe71b7fe640 10 monclient: dump: 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:24.640 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:24.641 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:24.641 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:24.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe722ec0640 5 monclient: authenticate success, global_id 4266 2024-03-29T11:35:24.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.640+0000 7fe722ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:24.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.641+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe714001b70 con 0x7fe71c0b17d0 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.641+0000 7fe71b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.641+0000 7fe71b7fe640 1 -- 172.21.15.22:0/2959170747 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe714001f00 con 0x7fe71c0b17d0 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.641+0000 7fe71b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.641+0000 7fe71b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.641+0000 7fe71b7fe640 10 monclient: dump: 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:24.642 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.641+0000 7fe722ec0640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.641+0000 7fe722ec0640 4 set_mon_vals no callback set 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.642+0000 7fe722ec0640 10 monclient: shutdown 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.642+0000 7fe722ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.642+0000 7fe722ec0640 1 -- 172.21.15.22:0/2959170747 >> v1:172.21.15.22:6789/0 conn(0x7fe71c0b17d0 legacy=0x7fe71c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.643+0000 7fe722ec0640 1 -- 172.21.15.22:0/2959170747 shutdown_connections 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.643+0000 7fe722ec0640 1 -- 172.21.15.22:0/2959170747 >> 172.21.15.22:0/2959170747 conn(0x7fe71c066a00 msgr2=0x7fe71c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:24.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.645+0000 7fe722ec0640 1 -- 172.21.15.22:0/2959170747 shutdown_connections 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.647+0000 7fe722ec0640 1 -- 172.21.15.22:0/2959170747 wait complete. 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.648+0000 7fe722ec0640 10 monclient: build_initial_monmap 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.648+0000 7fe722ec0640 10 monclient: monmap: 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:24.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.648+0000 7fe722ec0640 1 librados: starting msgr at 2024-03-29T11:35:24.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.648+0000 7fe722ec0640 1 librados: starting objecter 2024-03-29T11:35:24.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.649+0000 7fe722ec0640 1 Processor -- start 2024-03-29T11:35:24.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.649+0000 7fe722ec0640 1 -- start start 2024-03-29T11:35:24.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.649+0000 7fe722ec0640 1 librados: setting wanted keys 2024-03-29T11:35:24.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.649+0000 7fe722ec0640 1 librados: calling monclient init 2024-03-29T11:35:24.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.649+0000 7fe722ec0640 10 monclient: init 2024-03-29T11:35:24.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.651+0000 7fe722ec0640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:24.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.651+0000 7fe722ec0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:24.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.651+0000 7fe722ec0640 10 monclient(hunting): picked mon.noname-a con 0x7fe71c0b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:24.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.652+0000 7fe722ec0640 10 monclient(hunting): picked mon.noname-c con 0x7fe71c1422b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:24.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.652+0000 7fe722ec0640 10 monclient(hunting): picked mon.noname-b con 0x7fe71c142690 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:24.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.652+0000 7fe722ec0640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c0abf00 con 0x7fe71c0b17d0 2024-03-29T11:35:24.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.652+0000 7fe722ec0640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c0acda0 con 0x7fe71c142690 2024-03-29T11:35:24.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.652+0000 7fe722ec0640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe71c0ad080 con 0x7fe71c1422b0 2024-03-29T11:35:24.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.652+0000 7fe722ec0640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:24.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.652+0000 7fe722ec0640 10 monclient(hunting): authenticate will time out at 974.929556s 2024-03-29T11:35:24.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.657+0000 7fe720c5c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fe71c0b17d0 0x7fe71c12ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:47591/0 (socket says 172.21.15.22:47591) 2024-03-29T11:35:24.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.657+0000 7fe720c5c640 1 -- 172.21.15.22:0/966874853 learned_addr learned my addr 172.21.15.22:0/966874853 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:24.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 10 client.?.objecter ms_handle_connect 0x7fe71c1422b0 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 10 client.?.objecter ms_handle_connect 0x7fe71c142690 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127763647 0 0) 0x7fe71c0ad080 con 0x7fe71c1422b0 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.659+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe708004eb0 con 0x7fe71c1422b0 2024-03-29T11:35:24.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.660+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1884371346 0 0) 0x7fe708004eb0 con 0x7fe71c1422b0 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.660+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe71c0ad080 con 0x7fe71c1422b0 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.660+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe7100017d0 con 0x7fe71c1422b0 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.660+0000 7fe7197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.660+0000 7fe7197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.660+0000 7fe7197fa640 10 monclient(hunting): dump: 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:24.660 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:24.661 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:24.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.660+0000 7fe7197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756055725 0 0) 0x7fe71c0acda0 con 0x7fe71c142690 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe708004eb0 con 0x7fe71c142690 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 client.?.objecter ms_handle_connect 0x7fe71c0b17d0 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597814013 0 0) 0x7fe71c0abf00 con 0x7fe71c0b17d0 2024-03-29T11:35:24.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:24.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:24.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:24.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:35:24.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.661+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c0acda0 con 0x7fe71c0b17d0 2024-03-29T11:35:24.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.662+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1506657448 0 0) 0x7fe71c0ad080 con 0x7fe71c1422b0 2024-03-29T11:35:24.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 >> v1:172.21.15.204:6789/0 conn(0x7fe71c142690 legacy=0x7fe71c1e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:24.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 >> v1:172.21.15.22:6789/0 conn(0x7fe71c0b17d0 legacy=0x7fe71c12ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:24.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:24.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 1 monclient: found mon.c 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c132390 con 0x7fe71c1422b0 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe7197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 10 monclient: _renew_subs 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71c0aba00 con 0x7fe71c1422b0 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 10 monclient: _renew_subs 2024-03-29T11:35:24.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:24.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71c131fe0 con 0x7fe71c1422b0 2024-03-29T11:35:24.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.663+0000 7fe722ec0640 1 librados: init done 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.666+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe710001d60 con 0x7fe71c1422b0 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.666+0000 7fe7197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.666+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe710003790 con 0x7fe71c1422b0 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.666+0000 7fe7197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.666+0000 7fe7197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.666+0000 7fe7197fa640 10 monclient: dump: 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:24.668 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.666+0000 7fe71a7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.667+0000 7fe722ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.667+0000 7fe722ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.667+0000 7fe722ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.668+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc005080 con 0x7fe71c1422b0 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.669+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fe7100038f0 con 0x7fe71c1422b0 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.669+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fe710083810 con 0x7fe71c1422b0 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.669+0000 7fe7197fa640 10 client.4253.objecter ms_dispatch 0x7fe71c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:24.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.669+0000 7fe7197fa640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:24.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.669+0000 7fe7197fa640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:24.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.669+0000 7fe7197fa640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:35:24.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.670+0000 7fe720c5c640 10 monclient: get_auth_request con 0x7fe70806dea0 auth_method 0 2024-03-29T11:35:24.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.670+0000 7fe7197fa640 10 client.4253.objecter ms_handle_connect 0x7fe70806dea0 2024-03-29T11:35:24.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.677+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe71004ef20 con 0x7fe71c1422b0 2024-03-29T11:35:24.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.677+0000 7fe7197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:24.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:24.677+0000 7fe7197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:25.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.002+0000 7fe722ec0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.002+0000 7fe722ec0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.002+0000 7fe722ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:25.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.002+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6fc005330 con 0x7fe71c1422b0 2024-03-29T11:35:25.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.005+0000 7fe7197fa640 1 -- 172.21.15.22:0/966874853 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fe710051020 con 0x7fe71c1422b0 2024-03-29T11:35:25.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.005+0000 7fe7197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.005+0000 7fe7197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:25.005 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:35:25.005 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:25.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.008+0000 7fe722ec0640 10 librados: watch_flush enter 2024-03-29T11:35:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.010+0000 7fe722ec0640 10 librados: watch_flush exit 2024-03-29T11:35:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.010+0000 7fe722ec0640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:35:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.010+0000 7fe722ec0640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:35:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.010+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 >> v1:172.21.15.204:6816/3848099522 conn(0x7fe70806dea0 legacy=0x7fe708070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.010+0000 7fe722ec0640 10 monclient: shutdown 2024-03-29T11:35:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.010+0000 7fe722ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.010+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 >> v1:172.21.15.22:6790/0 conn(0x7fe71c1422b0 legacy=0x7fe71c1306f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.011+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 shutdown_connections 2024-03-29T11:35:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.011+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 >> 172.21.15.22:0/966874853 conn(0x7fe71c066a00 msgr2=0x7fe71c13dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.011+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 shutdown_connections 2024-03-29T11:35:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.011+0000 7fe722ec0640 1 -- 172.21.15.22:0/966874853 wait complete. 2024-03-29T11:35:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:25.012+0000 7fe722ec0640 1 librados: shutdown 2024-03-29T11:35:25.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:31.835 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:31.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.984+0000 7f4f82c34640 10 monclient: get_monmap_and_config 2024-03-29T11:35:31.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.984+0000 7f4f82c34640 10 monclient: build_initial_monmap 2024-03-29T11:35:31.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.984+0000 7f4f82c34640 10 monclient: monmap: 2024-03-29T11:35:31.983 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:31.983 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:31.983 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:35:31.986211+0000 2024-03-29T11:35:31.984 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:35:31.986211+0000 2024-03-29T11:35:31.984 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:31.984 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:31.984 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:31.984 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:35:31.984 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:35:31.984 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.986+0000 7f4f82c34640 1 Processor -- start 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.986+0000 7f4f82c34640 1 -- start start 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.986+0000 7f4f82c34640 10 monclient: init 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 10 monclient(hunting): picked mon.noname-b con 0x7f4f74006e30 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 10 monclient(hunting): picked mon.noname-a con 0x7f4f74035e00 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 10 monclient(hunting): picked mon.noname-c con 0x7f4f740c9670 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f74035970 con 0x7f4f74035e00 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f74036280 con 0x7f4f740c9670 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f74035640 con 0x7f4f74006e30 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:31.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.987+0000 7f4f82c34640 10 monclient(hunting): authenticate will time out at 982.264502s 2024-03-29T11:35:31.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.991+0000 7f4f81c32640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4f74006e30 0x7f4f740cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:43939/0 (socket says 172.21.15.22:43939) 2024-03-29T11:35:31.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.991+0000 7f4f81c32640 1 -- 172.21.15.22:0/2200660253 learned_addr learned my addr 172.21.15.22:0/2200660253 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:31.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.994+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226086268 0 0) 0x7f4f74036280 con 0x7f4f740c9670 2024-03-29T11:35:31.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.994+0000 7f4f80c30640 10 monclient(hunting): _negotiate 2024-03-29T11:35:31.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.994+0000 7f4f80c30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:31.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.994+0000 7f4f80c30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:31.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.994+0000 7f4f80c30640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:35:31.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.994+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f68003520 con 0x7f4f740c9670 2024-03-29T11:35:31.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628117965 0 0) 0x7f4f74035640 con 0x7f4f74006e30 2024-03-29T11:35:31.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient(hunting): _negotiate 2024-03-29T11:35:31.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:31.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:35:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f68003840 con 0x7f4f74006e30 2024-03-29T11:35:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719299061 0 0) 0x7f4f74035970 con 0x7f4f74035e00 2024-03-29T11:35:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient(hunting): _negotiate 2024-03-29T11:35:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:35:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f74035640 con 0x7f4f74035e00 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3166088935 0 0) 0x7f4f74035640 con 0x7f4f74035e00 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 >> v1:172.21.15.22:6790/0 conn(0x7f4f74006e30 legacy=0x7f4f740cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 >> v1:172.21.15.204:6789/0 conn(0x7f4f740c9670 legacy=0x7f4f740d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient: _finish_hunting 0 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 monclient: found mon.noname-a 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74034e20 con 0x7f4f74035e00 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient: _finish_auth 0 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient: _check_auth_tickets 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f7c04f730 con 0x7f4f74035e00 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.996+0000 7f4f80c30640 10 monclient: dump: 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f7c0008d0 con 0x7f4f74035e00 2024-03-29T11:35:31.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f80c30640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f80c30640 1 -- 172.21.15.22:0/2200660253 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f7c04f730 con 0x7f4f74035e00 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f80c30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f80c30640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f80c30640 10 monclient: dump: 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:31.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f82c34640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:35:31.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f82c34640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:31.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.997+0000 7f4f82c34640 4 set_mon_vals no callback set 2024-03-29T11:35:31.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.998+0000 7f4f82c34640 10 monclient: shutdown 2024-03-29T11:35:31.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.998+0000 7f4f82c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:31.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.998+0000 7f4f82c34640 1 -- 172.21.15.22:0/2200660253 >> v1:172.21.15.22:6789/0 conn(0x7f4f74035e00 legacy=0x7f4f740c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.999+0000 7f4f82c34640 1 -- 172.21.15.22:0/2200660253 shutdown_connections 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.999+0000 7f4f82c34640 1 -- 172.21.15.22:0/2200660253 >> 172.21.15.22:0/2200660253 conn(0x7f4f74005c00 msgr2=0x7f4f74006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.999+0000 7f4f82c34640 1 -- 172.21.15.22:0/2200660253 shutdown_connections 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:31.999+0000 7f4f82c34640 1 -- 172.21.15.22:0/2200660253 wait complete. 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.000+0000 7f4f82c34640 10 monclient: build_initial_monmap 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.000+0000 7f4f82c34640 10 monclient: monmap: 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:32.001 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.001+0000 7f4f82c34640 1 librados: starting msgr at 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.001+0000 7f4f82c34640 1 librados: starting objecter 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.001+0000 7f4f82c34640 1 Processor -- start 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.001+0000 7f4f82c34640 1 -- start start 2024-03-29T11:35:32.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.001+0000 7f4f82c34640 1 librados: setting wanted keys 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.001+0000 7f4f82c34640 1 librados: calling monclient init 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.001+0000 7f4f82c34640 10 monclient: init 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 10 monclient(hunting): picked mon.noname-b con 0x7f4f74006e30 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 10 monclient(hunting): picked mon.noname-a con 0x7f4f74035e00 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 10 monclient(hunting): picked mon.noname-c con 0x7f4f740c9670 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f740bca60 con 0x7f4f74035e00 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7402e890 con 0x7f4f74006e30 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f740bd650 con 0x7f4f740c9670 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:32.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.002+0000 7f4f82c34640 10 monclient(hunting): authenticate will time out at 982.279325s 2024-03-29T11:35:32.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.008+0000 7f4f81c32640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f4f74006e30 0x7f4f740bcf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:60201/0 (socket says 172.21.15.22:60201) 2024-03-29T11:35:32.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.008+0000 7f4f81c32640 1 -- 172.21.15.22:0/2742276408 learned_addr learned my addr 172.21.15.22:0/2742276408 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:32.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.009+0000 7f4f727fc640 10 client.?.objecter ms_handle_connect 0x7f4f74006e30 2024-03-29T11:35:32.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.009+0000 7f4f727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.010+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596278149 0 0) 0x7f4f7402e890 con 0x7f4f74006e30 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.010+0000 7f4f727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.010+0000 7f4f727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.010+0000 7f4f727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.010+0000 7f4f727fc640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.010+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f60004eb0 con 0x7f4f74006e30 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.010+0000 7f4f727fc640 10 client.?.objecter ms_handle_connect 0x7f4f740c9670 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.010+0000 7f4f727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.011+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991997833 0 0) 0x7f4f740bd650 con 0x7f4f740c9670 2024-03-29T11:35:32.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.011+0000 7f4f727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.011+0000 7f4f727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.011+0000 7f4f727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.011+0000 7f4f727fc640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:35:32.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.011+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f7402e890 con 0x7f4f740c9670 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 10 client.?.objecter ms_handle_connect 0x7f4f74035e00 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 274990047 0 0) 0x7f4f7402e890 con 0x7f4f740c9670 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f740bd650 con 0x7f4f740c9670 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f6c001790 con 0x7f4f740c9670 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 10 monclient(hunting): dump: 2024-03-29T11:35:32.011 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:32.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.012+0000 7f4f727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1066738547 0 0) 0x7f4f740bd650 con 0x7f4f740c9670 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 >> v1:172.21.15.204:6789/0 conn(0x7f4f74006e30 legacy=0x7f4f740bcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 >> v1:172.21.15.22:6789/0 conn(0x7f4f74035e00 legacy=0x7f4f740b9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 1 monclient: found mon.c 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f740bd7e0 con 0x7f4f740c9670 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 10 monclient: _renew_subs 2024-03-29T11:35:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f74033be0 con 0x7f4f740c9670 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 10 monclient: _renew_subs 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f740ba540 con 0x7f4f740c9670 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.013+0000 7f4f82c34640 1 librados: init done 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.015+0000 7f4f82c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.015+0000 7f4f82c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.015+0000 7f4f82c34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:32.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.016+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f74007b70 con 0x7f4f740c9670 2024-03-29T11:35:32.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.016+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f6c001c80 con 0x7f4f740c9670 2024-03-29T11:35:32.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.016+0000 7f4f727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:32.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.016+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f6c003790 con 0x7f4f740c9670 2024-03-29T11:35:32.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.016+0000 7f4f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.016+0000 7f4f727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.016+0000 7f4f727fc640 10 monclient: dump: 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:32.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.016+0000 7f4f73fff640 4 set_mon_vals no callback set 2024-03-29T11:35:32.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.018+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f4f6c019760 con 0x7f4f740c9670 2024-03-29T11:35:32.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.022+0000 7f4f81c32640 10 monclient: get_auth_request con 0x7f4f6006dcc0 auth_method 0 2024-03-29T11:35:32.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.022+0000 7f4f727fc640 10 client.4259.objecter ms_handle_connect 0x7f4f6006dcc0 2024-03-29T11:35:32.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.022+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f4f6c053420 con 0x7f4f740c9670 2024-03-29T11:35:32.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.022+0000 7f4f727fc640 10 client.4259.objecter ms_dispatch 0x7f4f7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:32.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.022+0000 7f4f727fc640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:32.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.022+0000 7f4f727fc640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:32.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.022+0000 7f4f727fc640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T11:35:32.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.029+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f6c04f6c0 con 0x7f4f740c9670 2024-03-29T11:35:32.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.029+0000 7f4f727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.029+0000 7f4f727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:32.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.358+0000 7f4f82c34640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.358+0000 7f4f82c34640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.358+0000 7f4f82c34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:32.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.358+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f74005120 con 0x7f4f740c9670 2024-03-29T11:35:32.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.360+0000 7f4f727fc640 1 -- 172.21.15.22:0/2742276408 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f4f6c04f860 con 0x7f4f740c9670 2024-03-29T11:35:32.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.360+0000 7f4f727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.360+0000 7f4f727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:32.362 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:35:32.362 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:32.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.366+0000 7f4f82c34640 10 librados: watch_flush enter 2024-03-29T11:35:32.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 10 librados: watch_flush exit 2024-03-29T11:35:32.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T11:35:32.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 10 client.4259.objecter successfully canceled tick 2024-03-29T11:35:32.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 >> v1:172.21.15.204:6816/3848099522 conn(0x7f4f6006dcc0 legacy=0x7f4f60070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 10 monclient: shutdown 2024-03-29T11:35:32.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:32.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 >> v1:172.21.15.22:6790/0 conn(0x7f4f740c9670 legacy=0x7f4f740b9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 shutdown_connections 2024-03-29T11:35:32.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.367+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 >> 172.21.15.22:0/2742276408 conn(0x7f4f74005c00 msgr2=0x7f4f740c8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:32.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.368+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 shutdown_connections 2024-03-29T11:35:32.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.368+0000 7f4f82c34640 1 -- 172.21.15.22:0/2742276408 wait complete. 2024-03-29T11:35:32.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:32.369+0000 7f4f82c34640 1 librados: shutdown 2024-03-29T11:35:32.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:39.190 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:39.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.343+0000 7fd9191ad640 10 monclient: get_monmap_and_config 2024-03-29T11:35:39.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.343+0000 7fd9191ad640 10 monclient: build_initial_monmap 2024-03-29T11:35:39.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.343+0000 7fd9191ad640 10 monclient: monmap: 2024-03-29T11:35:39.342 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:39.342 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:39.342 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:35:39.344948+0000 2024-03-29T11:35:39.342 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:35:39.344948+0000 2024-03-29T11:35:39.343 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:39.343 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:39.343 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:39.343 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:35:39.343 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:35:39.343 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:39.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.344+0000 7fd9191ad640 1 Processor -- start 2024-03-29T11:35:39.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.345+0000 7fd9191ad640 1 -- start start 2024-03-29T11:35:39.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.345+0000 7fd9191ad640 10 monclient: init 2024-03-29T11:35:39.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.345+0000 7fd9191ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:39.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.345+0000 7fd9191ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:39.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.345+0000 7fd9191ad640 10 monclient(hunting): picked mon.noname-a con 0x7fd90c006a70 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:39.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9191ad640 10 monclient(hunting): picked mon.noname-c con 0x7fd90c02b100 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:39.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9191ad640 10 monclient(hunting): picked mon.noname-b con 0x7fd90c029c30 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:39.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9191ad640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c02a270 con 0x7fd90c006a70 2024-03-29T11:35:39.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9191ad640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c02c440 con 0x7fd90c02b100 2024-03-29T11:35:39.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9189ac640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fd90c029c30 0x7fd90c0d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45752/0 (socket says 172.21.15.22:45752) 2024-03-29T11:35:39.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9189ac640 1 -- 172.21.15.22:0/3433163885 learned_addr learned my addr 172.21.15.22:0/3433163885 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:39.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9191ad640 1 -- 172.21.15.22:0/3433163885 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0c9440 con 0x7fd90c029c30 2024-03-29T11:35:39.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9191ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:39.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd9191ad640 10 monclient(hunting): authenticate will time out at 989.623404s 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67113569 0 0) 0x7fd90c0c9440 con 0x7fd90c029c30 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd904003520 con 0x7fd90c029c30 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355810859 0 0) 0x7fd90c02a270 con 0x7fd90c006a70 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.346+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90c0c9440 con 0x7fd90c006a70 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3486247889 0 0) 0x7fd904003520 con 0x7fd90c029c30 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 >> v1:172.21.15.204:6789/0 conn(0x7fd90c02b100 legacy=0x7fd90c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 >> v1:172.21.15.22:6789/0 conn(0x7fd90c006a70 legacy=0x7fd90c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 1 monclient: found mon.noname-b 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c0c95d0 con 0x7fd90c029c30 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd91404f730 con 0x7fd90c029c30 2024-03-29T11:35:39.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd912ffd640 10 monclient: dump: 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd9191ad640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:35:39.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.347+0000 7fd9191ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:39.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.348+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9140008d0 con 0x7fd90c029c30 2024-03-29T11:35:39.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.348+0000 7fd912ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:39.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.348+0000 7fd912ffd640 1 -- 172.21.15.22:0/3433163885 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd9140758b0 con 0x7fd90c029c30 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.348+0000 7fd912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.348+0000 7fd912ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.348+0000 7fd912ffd640 10 monclient: dump: 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:39.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.348+0000 7fd9191ad640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.348+0000 7fd9191ad640 4 set_mon_vals no callback set 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.349+0000 7fd9191ad640 10 monclient: shutdown 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.349+0000 7fd9191ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.349+0000 7fd9191ad640 1 -- 172.21.15.22:0/3433163885 >> v1:172.21.15.22:6790/0 conn(0x7fd90c029c30 legacy=0x7fd90c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.349+0000 7fd9191ad640 1 -- 172.21.15.22:0/3433163885 shutdown_connections 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.349+0000 7fd9191ad640 1 -- 172.21.15.22:0/3433163885 >> 172.21.15.22:0/3433163885 conn(0x7fd90c0058f0 msgr2=0x7fd90c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.349+0000 7fd9191ad640 1 -- 172.21.15.22:0/3433163885 shutdown_connections 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.349+0000 7fd9191ad640 1 -- 172.21.15.22:0/3433163885 wait complete. 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.350+0000 7fd9191ad640 10 monclient: build_initial_monmap 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.350+0000 7fd9191ad640 10 monclient: monmap: 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:35:39.353 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.350+0000 7fd9191ad640 1 librados: starting msgr at 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.350+0000 7fd9191ad640 1 librados: starting objecter 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.351+0000 7fd9191ad640 1 Processor -- start 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.351+0000 7fd9191ad640 1 -- start start 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.351+0000 7fd9191ad640 1 librados: setting wanted keys 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.351+0000 7fd9191ad640 1 librados: calling monclient init 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.351+0000 7fd9191ad640 10 monclient: init 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.352+0000 7fd9191ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:39.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.352+0000 7fd9191ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.352+0000 7fd9191ad640 10 monclient(hunting): picked mon.noname-b con 0x7fd90c006a70 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.352+0000 7fd9191ad640 10 monclient(hunting): picked mon.noname-a con 0x7fd90c029c30 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd913fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fd90c006a70 0x7fd90c0ca470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:39544/0 (socket says 172.21.15.22:39544) 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd9191ad640 10 monclient(hunting): picked mon.noname-c con 0x7fd90c0cab80 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd913fff640 1 -- 172.21.15.22:0/2495630399 learned_addr learned my addr 172.21.15.22:0/2495630399 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 10 client.?.objecter ms_handle_connect 0x7fd90c029c30 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 10 client.?.objecter ms_handle_connect 0x7fd90c0cab80 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0cdd90 con 0x7fd90c029c30 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0c04e0 con 0x7fd90c006a70 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 10 client.?.objecter ms_handle_connect 0x7fd90c006a70 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:39.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0cf140 con 0x7fd90c0cab80 2024-03-29T11:35:39.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd9191ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:39.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd9191ad640 10 monclient(hunting): authenticate will time out at 989.630593s 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123162220 0 0) 0x7fd90c0cf140 con 0x7fd90c0cab80 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.353+0000 7fd910ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f8004eb0 con 0x7fd90c0cab80 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458467957 0 0) 0x7fd90c0cdd90 con 0x7fd90c029c30 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90c0cf140 con 0x7fd90c029c30 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095164437 0 0) 0x7fd90c0c04e0 con 0x7fd90c006a70 2024-03-29T11:35:39.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90c0cdd90 con 0x7fd90c006a70 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 404702989 0 0) 0x7fd8f8004eb0 con 0x7fd90c0cab80 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd90c0c04e0 con 0x7fd90c0cab80 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd91404f400 con 0x7fd90c0cab80 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:39.362 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.354+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 951935872 0 0) 0x7fd90c0cf140 con 0x7fd90c029c30 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8f8004eb0 con 0x7fd90c029c30 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2982825132 0 0) 0x7fd90c0cdd90 con 0x7fd90c006a70 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd90c0cf140 con 0x7fd90c006a70 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd908001510 con 0x7fd90c029c30 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd900001870 con 0x7fd90c006a70 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1952127662 0 0) 0x7fd90c0c04e0 con 0x7fd90c0cab80 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 >> v1:172.21.15.204:6789/0 conn(0x7fd90c006a70 legacy=0x7fd90c0ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 >> v1:172.21.15.22:6789/0 conn(0x7fd90c029c30 legacy=0x7fd90c0bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:39.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 monclient: found mon.c 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c02c940 con 0x7fd90c0cab80 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd910ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 10 monclient: _renew_subs 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c0ce220 con 0x7fd90c0cab80 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 10 monclient: _renew_subs 2024-03-29T11:35:39.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c0cd210 con 0x7fd90c0cab80 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.355+0000 7fd9191ad640 1 librados: init done 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.356+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd91404fdd0 con 0x7fd90c0cab80 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.356+0000 7fd910ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.356+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd914067bf0 con 0x7fd90c0cab80 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.356+0000 7fd9127fc640 4 set_mon_vals no callback set 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.356+0000 7fd910ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.356+0000 7fd910ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.356+0000 7fd910ff9640 10 monclient: dump: 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.367 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.357+0000 7fd9191ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.357+0000 7fd9191ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.357+0000 7fd9191ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.357+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f4005080 con 0x7fd90c0cab80 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.359+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fd914067d50 con 0x7fd90c0cab80 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.360+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fd9140ff340 con 0x7fd90c0cab80 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.360+0000 7fd910ff9640 10 client.4265.objecter ms_dispatch 0x7fd914000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.360+0000 7fd910ff9640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.360+0000 7fd910ff9640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:39.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.360+0000 7fd910ff9640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:35:39.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.360+0000 7fd913fff640 10 monclient: get_auth_request con 0x7fd8f806def0 auth_method 0 2024-03-29T11:35:39.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.360+0000 7fd910ff9640 10 client.4265.objecter ms_handle_connect 0x7fd8f806def0 2024-03-29T11:35:39.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.368+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9140cac80 con 0x7fd90c0cab80 2024-03-29T11:35:39.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.368+0000 7fd910ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.368+0000 7fd910ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:39.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.737+0000 7fd9191ad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:39.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.737+0000 7fd9191ad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:39.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.737+0000 7fd9191ad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:39.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.737+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8f4005330 con 0x7fd90c0cab80 2024-03-29T11:35:39.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.738+0000 7fd910ff9640 1 -- 172.21.15.22:0/2495630399 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fd9140fe9f0 con 0x7fd90c0cab80 2024-03-29T11:35:39.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.738+0000 7fd910ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:39.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.738+0000 7fd910ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:39.737 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:35:39.737 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:39.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.741+0000 7fd9191ad640 10 librados: watch_flush enter 2024-03-29T11:35:39.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.741+0000 7fd9191ad640 10 librados: watch_flush exit 2024-03-29T11:35:39.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.741+0000 7fd9191ad640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:35:39.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.741+0000 7fd9191ad640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:35:39.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.742+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 >> v1:172.21.15.204:6816/3848099522 conn(0x7fd8f806def0 legacy=0x7fd8f80703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.742+0000 7fd9191ad640 10 monclient: shutdown 2024-03-29T11:35:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.742+0000 7fd9191ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.742+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 >> v1:172.21.15.22:6790/0 conn(0x7fd90c0cab80 legacy=0x7fd90c0bfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.742+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 shutdown_connections 2024-03-29T11:35:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.742+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 >> 172.21.15.22:0/2495630399 conn(0x7fd90c0058f0 msgr2=0x7fd90c0c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.742+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 shutdown_connections 2024-03-29T11:35:39.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.743+0000 7fd9191ad640 1 -- 172.21.15.22:0/2495630399 wait complete. 2024-03-29T11:35:39.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:39.743+0000 7fd9191ad640 1 librados: shutdown 2024-03-29T11:35:39.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:46.563 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.719+0000 7f376759e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.719+0000 7f376759e640 10 monclient: build_initial_monmap 2024-03-29T11:35:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.719+0000 7f376759e640 10 monclient: monmap: 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:35:46.720195+0000 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:35:46.720195+0000 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:35:46.718 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.720+0000 7f376759e640 1 Processor -- start 2024-03-29T11:35:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.721+0000 7f376759e640 1 -- start start 2024-03-29T11:35:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.721+0000 7f376759e640 10 monclient: init 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 10 monclient(hunting): picked mon.noname-b con 0x7f37680b2790 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 10 monclient(hunting): picked mon.noname-c con 0x7f3768143c80 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 10 monclient(hunting): picked mon.noname-a con 0x7f3768146e30 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3768146290 con 0x7f3768146e30 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37680b3970 con 0x7f3768143c80 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376806ee80 con 0x7f37680b2790 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376759e640 10 monclient(hunting): authenticate will time out at 996.998464s 2024-03-29T11:35:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376659c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f37680b2790 0x7f3768143570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55004/0 (socket says 172.21.15.22:55004) 2024-03-29T11:35:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.722+0000 7f376659c640 1 -- 172.21.15.22:0/4260312430 learned_addr learned my addr 172.21.15.22:0/4260312430 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364163090 0 0) 0x7f37680b3970 con 0x7f3768143c80 2024-03-29T11:35:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:35:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3750003520 con 0x7f3768143c80 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2366995440 0 0) 0x7f3750003520 con 0x7f3768143c80 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 >> v1:172.21.15.22:6790/0 conn(0x7f37680b2790 legacy=0x7f3768143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 >> v1:172.21.15.22:6789/0 conn(0x7f3768146e30 legacy=0x7f3768151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 1 monclient: found mon.noname-c 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376806f160 con 0x7f3768143c80 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 10 monclient: _finish_auth 0 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.723+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3758001670 con 0x7f3768143c80 2024-03-29T11:35:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 10 monclient: dump: 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376759e640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:35:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:46.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3758001850 con 0x7f3768143c80 2024-03-29T11:35:46.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376759e640 4 set_mon_vals no callback set 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 1 -- 172.21.15.22:0/4260312430 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3758001670 con 0x7f3768143c80 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.724+0000 7f376559a640 10 monclient: dump: 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:46.725 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.725+0000 7f376759e640 10 monclient: shutdown 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.725+0000 7f376759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.725+0000 7f376759e640 1 -- 172.21.15.22:0/4260312430 >> v1:172.21.15.204:6789/0 conn(0x7f3768143c80 legacy=0x7f3768146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.725+0000 7f376759e640 1 -- 172.21.15.22:0/4260312430 shutdown_connections 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.725+0000 7f376759e640 1 -- 172.21.15.22:0/4260312430 >> 172.21.15.22:0/4260312430 conn(0x7f376806d070 msgr2=0x7f3768066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.725+0000 7f376759e640 1 -- 172.21.15.22:0/4260312430 shutdown_connections 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.726+0000 7f376759e640 1 -- 172.21.15.22:0/4260312430 wait complete. 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.727+0000 7f376759e640 10 monclient: build_initial_monmap 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.727+0000 7f376759e640 10 monclient: monmap: 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:46.726 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:46.727 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:35:46.727 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:35:46.727 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:46.727 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:46.727 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:46.727 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:35:46.727 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:35:46.727 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.727+0000 7f376759e640 1 librados: starting msgr at 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.727+0000 7f376759e640 1 librados: starting objecter 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.728+0000 7f376759e640 1 Processor -- start 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.728+0000 7f376759e640 1 -- start start 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.728+0000 7f376759e640 1 librados: setting wanted keys 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.728+0000 7f376759e640 1 librados: calling monclient init 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.728+0000 7f376759e640 10 monclient: init 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f376759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f376759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f376759e640 10 monclient(hunting): picked mon.noname-b con 0x7f37680b2790 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f376759e640 10 monclient(hunting): picked mon.noname-c con 0x7f3768143c80 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f376759e640 10 monclient(hunting): picked mon.noname-a con 0x7f3768146e30 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f3765d9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f3768143c80 0x7f376813a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55268/0 (socket says 172.21.15.22:55268) 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f3765d9b640 1 -- 172.21.15.22:0/2265249465 learned_addr learned my addr 172.21.15.22:0/2265249465 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f376759e640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3768149b60 con 0x7f3768146e30 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f376759e640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376813acf0 con 0x7f37680b2790 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.729+0000 7f3756ffd640 10 client.?.objecter ms_handle_connect 0x7f37680b2790 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 10 client.?.objecter ms_handle_connect 0x7f3768143c80 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f376759e640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376813ae80 con 0x7f3768143c80 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f376759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f376759e640 10 monclient(hunting): authenticate will time out at 997.006233s 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104552803 0 0) 0x7f376813acf0 con 0x7f37680b2790 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.730+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3748004eb0 con 0x7f37680b2790 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 client.?.objecter ms_handle_connect 0x7f3768146e30 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987615935 0 0) 0x7f376813ae80 con 0x7f3768143c80 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f376813acf0 con 0x7f3768143c80 2024-03-29T11:35:46.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516151244 0 0) 0x7f3768149b60 con 0x7f3768146e30 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f376813ae80 con 0x7f3768146e30 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2803083607 0 0) 0x7f3748004eb0 con 0x7f37680b2790 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3768149b60 con 0x7f37680b2790 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f375c0018a0 con 0x7f37680b2790 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:46.734 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.731+0000 7f3756ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 997522449 0 0) 0x7f376813acf0 con 0x7f3768143c80 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3748004eb0 con 0x7f3768143c80 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3758001670 con 0x7f3768143c80 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:46.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3218873068 0 0) 0x7f376813ae80 con 0x7f3768146e30 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f376813acf0 con 0x7f3768146e30 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3760001ad0 con 0x7f3768146e30 2024-03-29T11:35:46.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1381909283 0 0) 0x7f3768149b60 con 0x7f37680b2790 2024-03-29T11:35:46.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 >> v1:172.21.15.22:6790/0 conn(0x7f3768143c80 legacy=0x7f376813a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 >> v1:172.21.15.22:6789/0 conn(0x7f3768146e30 legacy=0x7f37681e71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 monclient: found mon.b 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376813b010 con 0x7f37680b2790 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f3756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 10 monclient: _renew_subs 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376813fe10 con 0x7f37680b2790 2024-03-29T11:35:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 10 monclient: _renew_subs 2024-03-29T11:35:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f376813a730 con 0x7f37680b2790 2024-03-29T11:35:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.732+0000 7f376759e640 1 librados: init done 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.734+0000 7f3754ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.734+0000 7f3754ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.734+0000 7f3754ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.734+0000 7f3754ff9640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376804f730 con 0x7f37680b2790 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.735+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f375c001d10 con 0x7f37680b2790 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.735+0000 7f3756ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.735+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f375c004980 con 0x7f37680b2790 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.735+0000 7f3757fff640 4 set_mon_vals no callback set 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.735+0000 7f3756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.735+0000 7f3756ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.735+0000 7f3756ffd640 10 monclient: dump: 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:46.741 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.736+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f375c01a990 con 0x7f37680b2790 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.737+0000 7f3765d9b640 10 monclient: get_auth_request con 0x7f374806e190 auth_method 0 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.737+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f375c084a30 con 0x7f37680b2790 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.737+0000 7f3756ffd640 10 client.4282.objecter ms_dispatch 0x7f3768000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.737+0000 7f3756ffd640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.737+0000 7f3756ffd640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:46.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.737+0000 7f3756ffd640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:35:46.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.737+0000 7f3756ffd640 10 client.4282.objecter ms_handle_connect 0x7f374806e190 2024-03-29T11:35:46.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.742+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f375c0501c0 con 0x7f37680b2790 2024-03-29T11:35:46.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.742+0000 7f3756ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:46.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:46.742+0000 7f3756ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:47.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.116+0000 7f3754ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.116+0000 7f3754ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.117+0000 7f3754ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:35:47.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.117+0000 7f3754ff9640 1 -- 172.21.15.22:0/2265249465 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37680717c0 con 0x7f37680b2790 2024-03-29T11:35:47.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.118+0000 7f3756ffd640 1 -- 172.21.15.22:0/2265249465 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f375c0522c0 con 0x7f37680b2790 2024-03-29T11:35:47.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.118+0000 7f3756ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.118+0000 7f3756ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:47.116 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:35:47.116 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:47.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.121+0000 7f3754ff9640 10 librados: watch_flush enter 2024-03-29T11:35:47.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.121+0000 7f3754ff9640 10 librados: watch_flush exit 2024-03-29T11:35:47.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.121+0000 7f3754ff9640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:35:47.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.121+0000 7f3754ff9640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:35:47.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.122+0000 7f3754ff9640 1 -- 172.21.15.22:0/2265249465 >> v1:172.21.15.204:6816/3848099522 conn(0x7f374806e190 legacy=0x7f3748070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.122+0000 7f3754ff9640 10 monclient: shutdown 2024-03-29T11:35:47.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.122+0000 7f3754ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:47.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.122+0000 7f3754ff9640 1 -- 172.21.15.22:0/2265249465 >> v1:172.21.15.204:6789/0 conn(0x7f37680b2790 legacy=0x7f3768149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.122+0000 7f3754ff9640 1 -- 172.21.15.22:0/2265249465 shutdown_connections 2024-03-29T11:35:47.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.122+0000 7f3754ff9640 1 -- 172.21.15.22:0/2265249465 >> 172.21.15.22:0/2265249465 conn(0x7f376806d070 msgr2=0x7f3768069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:47.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.122+0000 7f3754ff9640 1 -- 172.21.15.22:0/2265249465 shutdown_connections 2024-03-29T11:35:47.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.123+0000 7f3754ff9640 1 -- 172.21.15.22:0/2265249465 wait complete. 2024-03-29T11:35:47.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:47.123+0000 7f3754ff9640 1 librados: shutdown 2024-03-29T11:35:47.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:53.941 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:54.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.077+0000 7fe01010e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:54.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.077+0000 7fe01010e640 10 monclient: build_initial_monmap 2024-03-29T11:35:54.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.077+0000 7fe01010e640 10 monclient: monmap: 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:35:54.077385+0000 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:35:54.077385+0000 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:35:54.075 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:54.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.077+0000 7fe01010e640 1 Processor -- start 2024-03-29T11:35:54.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.078+0000 7fe01010e640 1 -- start start 2024-03-29T11:35:54.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.078+0000 7fe01010e640 10 monclient: init 2024-03-29T11:35:54.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0081422b0 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 10 monclient(hunting): picked mon.noname-b con 0x7fe008142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 10 monclient(hunting): picked mon.noname-a con 0x7fe0080b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008142c80 con 0x7fe0080b17d0 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0080aae40 con 0x7fe0081422b0 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00813fbd0 con 0x7fe008142690 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe01010e640 10 monclient(hunting): authenticate will time out at 1004.355497s 2024-03-29T11:35:54.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.079+0000 7fe00deaa640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fe0081422b0 0x7fe00813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:50622/0 (socket says 172.21.15.22:50622) 2024-03-29T11:35:54.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.080+0000 7fe00deaa640 1 -- 172.21.15.22:0/393959400 learned_addr learned my addr 172.21.15.22:0/393959400 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:54.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034432309 0 0) 0x7fe0080aae40 con 0x7fe0081422b0 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff4003520 con 0x7fe0081422b0 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353625170 0 0) 0x7fe00813fbd0 con 0x7fe008142690 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0080aae40 con 0x7fe008142690 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766852742 0 0) 0x7fdff4003520 con 0x7fe0081422b0 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 >> v1:172.21.15.22:6790/0 conn(0x7fe008142690 legacy=0x7fe0080b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.081+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 >> v1:172.21.15.22:6789/0 conn(0x7fe0080b17d0 legacy=0x7fe0080b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:54.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: _finish_hunting 0 2024-03-29T11:35:54.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 1 monclient: found mon.noname-c 2024-03-29T11:35:54.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:54.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:35:54.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe008151940 con 0x7fe0081422b0 2024-03-29T11:35:54.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: _finish_auth 0 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: _check_auth_tickets 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe0040015c0 con 0x7fe0081422b0 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: dump: 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:54.082 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe01010e640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe01010e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe004001720 con 0x7fe0081422b0 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 1 -- 172.21.15.22:0/393959400 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe004001ab0 con 0x7fe0081422b0 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe00cea8640 10 monclient: dump: 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.083 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.084 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:54.084 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:54.084 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:54.084 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:54.084 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:54.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe01010e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:54.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.083+0000 7fe01010e640 4 set_mon_vals no callback set 2024-03-29T11:35:54.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.085+0000 7fe01010e640 10 monclient: shutdown 2024-03-29T11:35:54.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.085+0000 7fe01010e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.085+0000 7fe01010e640 1 -- 172.21.15.22:0/393959400 >> v1:172.21.15.204:6789/0 conn(0x7fe0081422b0 legacy=0x7fe00813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.085+0000 7fe01010e640 1 -- 172.21.15.22:0/393959400 shutdown_connections 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.085+0000 7fe01010e640 1 -- 172.21.15.22:0/393959400 >> 172.21.15.22:0/393959400 conn(0x7fe008066a00 msgr2=0x7fe008066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.086+0000 7fe01010e640 1 -- 172.21.15.22:0/393959400 shutdown_connections 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.086+0000 7fe01010e640 1 -- 172.21.15.22:0/393959400 wait complete. 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.087+0000 7fe01010e640 10 monclient: build_initial_monmap 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.087+0000 7fe01010e640 10 monclient: monmap: 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:54.086 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:35:54.087 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:35:54.087 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:35:54.087 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:54.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.088+0000 7fe01010e640 1 librados: starting msgr at 2024-03-29T11:35:54.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.088+0000 7fe01010e640 1 librados: starting objecter 2024-03-29T11:35:54.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.088+0000 7fe01010e640 1 Processor -- start 2024-03-29T11:35:54.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.088+0000 7fe01010e640 1 -- start start 2024-03-29T11:35:54.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.088+0000 7fe01010e640 1 librados: setting wanted keys 2024-03-29T11:35:54.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.088+0000 7fe01010e640 1 librados: calling monclient init 2024-03-29T11:35:54.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.088+0000 7fe01010e640 10 monclient: init 2024-03-29T11:35:54.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.089+0000 7fe01010e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:54.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.089+0000 7fe01010e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:54.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.090+0000 7fe00deaa640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe0080b17d0 0x7fe0081d67e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49554/0 (socket says 172.21.15.22:49554) 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.090+0000 7fe00deaa640 1 -- 172.21.15.22:0/1220445182 learned_addr learned my addr 172.21.15.22:0/1220445182 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.090+0000 7fe01010e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0080b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.090+0000 7fdffe7fc640 10 client.?.objecter ms_handle_connect 0x7fe0080b17d0 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.090+0000 7fdffe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.091+0000 7fe01010e640 10 monclient(hunting): picked mon.noname-a con 0x7fe0081422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.091+0000 7fe01010e640 10 monclient(hunting): picked mon.noname-b con 0x7fe008142690 addr v1:172.21.15.204:6789/0 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.091+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0080b6e50 con 0x7fe0081422b0 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.091+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0080b70c0 con 0x7fe008142690 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.091+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0081d6ef0 con 0x7fe0080b17d0 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.091+0000 7fe01010e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:54.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.091+0000 7fe01010e640 10 monclient(hunting): authenticate will time out at 1004.367192s 2024-03-29T11:35:54.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.092+0000 7fdffe7fc640 10 client.?.objecter ms_handle_connect 0x7fe008142690 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.092+0000 7fdffe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828274452 0 0) 0x7fe0081d6ef0 con 0x7fe0080b17d0 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfec004eb0 con 0x7fe0080b17d0 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078093607 0 0) 0x7fe0080b70c0 con 0x7fe008142690 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:35:54.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0081d6ef0 con 0x7fe008142690 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 286831142 0 0) 0x7fe0081d6ef0 con 0x7fe008142690 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0080b70c0 con 0x7fe008142690 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4201640062 0 0) 0x7fdfec004eb0 con 0x7fe0080b17d0 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0081d6ef0 con 0x7fe0080b17d0 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe000001b30 con 0x7fe008142690 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.098 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe004001a30 con 0x7fe0080b17d0 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.099 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3192261650 0 0) 0x7fe0081d6ef0 con 0x7fe0080b17d0 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 >> v1:172.21.15.204:6789/0 conn(0x7fe008142690 legacy=0x7fe0081e2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.093+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 >> v1:172.21.15.22:6789/0 conn(0x7fe0081422b0 legacy=0x7fe00812dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 1 monclient: found mon.c 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0080b74e0 con 0x7fe0080b17d0 2024-03-29T11:35:54.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 10 monclient: _renew_subs 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe008141480 con 0x7fe0080b17d0 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 10 monclient: _renew_subs 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0080b6060 con 0x7fe0080b17d0 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fe01010e640 1 librados: init done 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe004004a90 con 0x7fe0080b17d0 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe004004da0 con 0x7fe0080b17d0 2024-03-29T11:35:54.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffe7fc640 10 monclient: dump: 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.094+0000 7fdffffff640 4 set_mon_vals no callback set 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fe00401ad70 con 0x7fe0080b17d0 2024-03-29T11:35:54.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fe004084b50 con 0x7fe0080b17d0 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fdffe7fc640 10 client.4277.objecter ms_dispatch 0x7fe008000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fdffe7fc640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fdffe7fc640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fdffe7fc640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fe00d6a9640 10 monclient: get_auth_request con 0x7fdfec06de40 auth_method 0 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fe01010e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fe01010e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fe01010e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.097+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0080b7670 con 0x7fe0080b17d0 2024-03-29T11:35:54.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.098+0000 7fdffe7fc640 10 client.4277.objecter ms_handle_connect 0x7fdfec06de40 2024-03-29T11:35:54.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.105+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0080b7670 con 0x7fe0080b17d0 2024-03-29T11:35:54.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.105+0000 7fdffe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.105+0000 7fdffe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:54.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.423+0000 7fe01010e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.423+0000 7fe01010e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.423+0000 7fe01010e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:35:54.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.423+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0080b7670 con 0x7fe0080b17d0 2024-03-29T11:35:54.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.424+0000 7fdffe7fc640 1 -- 172.21.15.22:0/1220445182 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fe0080b7670 con 0x7fe0080b17d0 2024-03-29T11:35:54.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.424+0000 7fdffe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.424+0000 7fdffe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:54.422 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:35:54.422 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:54.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.427+0000 7fe01010e640 10 librados: watch_flush enter 2024-03-29T11:35:54.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.427+0000 7fe01010e640 10 librados: watch_flush exit 2024-03-29T11:35:54.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.427+0000 7fe01010e640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-29T11:35:54.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.427+0000 7fe01010e640 10 client.4277.objecter successfully canceled tick 2024-03-29T11:35:54.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 >> v1:172.21.15.204:6816/3848099522 conn(0x7fdfec06de40 legacy=0x7fdfec070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 10 monclient: shutdown 2024-03-29T11:35:54.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:54.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 >> v1:172.21.15.22:6790/0 conn(0x7fe0080b17d0 legacy=0x7fe0081d67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 shutdown_connections 2024-03-29T11:35:54.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 >> 172.21.15.22:0/1220445182 conn(0x7fe008066a00 msgr2=0x7fe0080b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:54.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 shutdown_connections 2024-03-29T11:35:54.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 1 -- 172.21.15.22:0/1220445182 wait complete. 2024-03-29T11:35:54.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:35:54.428+0000 7fe01010e640 1 librados: shutdown 2024-03-29T11:35:54.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:01.246 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:01.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.385+0000 7f7fc78d9640 10 monclient: get_monmap_and_config 2024-03-29T11:36:01.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.385+0000 7f7fc78d9640 10 monclient: build_initial_monmap 2024-03-29T11:36:01.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.385+0000 7f7fc78d9640 10 monclient: monmap: 2024-03-29T11:36:01.384 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:36:01.386863+0000 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:36:01.386863+0000 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:36:01.385 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:01.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.387+0000 7f7fc78d9640 1 Processor -- start 2024-03-29T11:36:01.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.388+0000 7f7fc78d9640 1 -- start start 2024-03-29T11:36:01.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.388+0000 7f7fc78d9640 10 monclient: init 2024-03-29T11:36:01.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:01.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:01.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc01457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc013f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc00a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc00aaef0 con 0x7f7fc01457f0 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc0145bd0 con 0x7f7fc013f650 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc0145eb0 con 0x7f7fc00a9d70 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.389+0000 7f7fc78d9640 10 monclient(hunting): authenticate will time out at 1011.666880s 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.390+0000 7f7fc5e76640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f7fc00a9d70 0x7f7fc014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55262/0 (socket says 172.21.15.22:55262) 2024-03-29T11:36:01.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.390+0000 7f7fc5e76640 1 -- 172.21.15.22:0/178436339 learned_addr learned my addr 172.21.15.22:0/178436339 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.391+0000 7f7fb7fff640 1 -- 172.21.15.22:0/178436339 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444607011 0 0) 0x7f7fc0145eb0 con 0x7f7fc00a9d70 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.391+0000 7f7fb7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.391+0000 7f7fb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.391+0000 7f7fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.391+0000 7f7fb7fff640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.391+0000 7f7fb7fff640 1 -- 172.21.15.22:0/178436339 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fac003520 con 0x7f7fc00a9d70 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.392+0000 7f7fb7fff640 1 -- 172.21.15.22:0/178436339 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866016902 0 0) 0x7f7fac003520 con 0x7f7fc00a9d70 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.392+0000 7f7fb7fff640 1 -- 172.21.15.22:0/178436339 >> v1:172.21.15.204:6789/0 conn(0x7f7fc013f650 legacy=0x7f7fc00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:01.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.392+0000 7f7fb7fff640 1 -- 172.21.15.22:0/178436339 >> v1:172.21.15.22:6789/0 conn(0x7f7fc01457f0 legacy=0x7f7fc013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 1 monclient: found mon.noname-b 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 1 -- 172.21.15.22:0/178436339 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc0146340 con 0x7f7fc00a9d70 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 1 -- 172.21.15.22:0/178436339 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7fb8001a10 con 0x7f7fc00a9d70 2024-03-29T11:36:01.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fc78d9640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fc78d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 10 monclient: dump: 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:01.394 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:01.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fc78d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:01.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 1 -- 172.21.15.22:0/178436339 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fb8001be0 con 0x7f7fc00a9d70 2024-03-29T11:36:01.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:01.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fc78d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:01.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.393+0000 7f7fc78d9640 4 set_mon_vals no callback set 2024-03-29T11:36:01.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.394+0000 7f7fc78d9640 10 monclient: shutdown 2024-03-29T11:36:01.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.394+0000 7f7fc78d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:01.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.394+0000 7f7fc78d9640 1 -- 172.21.15.22:0/178436339 >> v1:172.21.15.22:6790/0 conn(0x7f7fc00a9d70 legacy=0x7f7fc014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.395+0000 7f7fc78d9640 1 -- 172.21.15.22:0/178436339 shutdown_connections 2024-03-29T11:36:01.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.395+0000 7f7fc78d9640 1 -- 172.21.15.22:0/178436339 >> 172.21.15.22:0/178436339 conn(0x7f7fc0066a00 msgr2=0x7f7fc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.396+0000 7f7fc78d9640 1 -- 172.21.15.22:0/178436339 shutdown_connections 2024-03-29T11:36:01.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.396+0000 7f7fc78d9640 1 -- 172.21.15.22:0/178436339 wait complete. 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 10 monclient: build_initial_monmap 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 10 monclient: monmap: 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:01.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 1 librados: starting msgr at 2024-03-29T11:36:01.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 1 librados: starting objecter 2024-03-29T11:36:01.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 1 Processor -- start 2024-03-29T11:36:01.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 1 -- start start 2024-03-29T11:36:01.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 1 librados: setting wanted keys 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 1 librados: calling monclient init 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.398+0000 7f7fc78d9640 10 monclient: init 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc00a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc013f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc01457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc006e660 con 0x7f7fc00a9d70 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc006e940 con 0x7f7fc01457f0 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc01dc1f0 con 0x7f7fc013f650 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc78d9640 10 monclient(hunting): authenticate will time out at 1011.676471s 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc4e74640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f7fc013f650 0x7f7fc01db3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55914/0 (socket says 172.21.15.22:55914) 2024-03-29T11:36:01.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fc4e74640 1 -- 172.21.15.22:0/1858311991 learned_addr learned my addr 172.21.15.22:0/1858311991 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fb5ffb640 10 client.?.objecter ms_handle_connect 0x7f7fc013f650 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569190997 0 0) 0x7f7fc01dc1f0 con 0x7f7fc013f650 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fb5ffb640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.399+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa4004eb0 con 0x7f7fc013f650 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2723104680 0 0) 0x7f7fa4004eb0 con 0x7f7fc013f650 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc01dc1f0 con 0x7f7fc013f650 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7fbc004080 con 0x7f7fc013f650 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:01.401 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 488183075 0 0) 0x7f7fc01dc1f0 con 0x7f7fc013f650 2024-03-29T11:36:01.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.400+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 >> v1:172.21.15.204:6789/0 conn(0x7f7fc01457f0 legacy=0x7f7fc01dbae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.402+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 >> v1:172.21.15.22:6789/0 conn(0x7f7fc00a9d70 legacy=0x7f7fc013aeb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fb5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fb5ffb640 1 monclient: found mon.c 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fb5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc0138a90 con 0x7f7fc013f650 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fb5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fb5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fc78d9640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fc78d9640 10 monclient: _renew_subs 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fc78d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.403+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc013b5c0 con 0x7f7fc013f650 2024-03-29T11:36:01.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.404+0000 7f7fc78d9640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.404+0000 7f7fc78d9640 10 monclient: _renew_subs 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.404+0000 7f7fc78d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.404+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc013a560 con 0x7f7fc013f650 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.404+0000 7f7fc78d9640 1 librados: init done 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.405+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fbc0044d0 con 0x7f7fc013f650 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.405+0000 7f7fb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.405+0000 7f7fb77fe640 4 set_mon_vals no callback set 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.406+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7fbc0042e0 con 0x7f7fc013f650 2024-03-29T11:36:01.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.406+0000 7f7fb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.406+0000 7f7fb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.406+0000 7f7fb5ffb640 10 monclient: dump: 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.407+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f7fbc0013e0 con 0x7f7fc013f650 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.407+0000 7f7fc78d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fc78d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:01.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fc78d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:01.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0005080 con 0x7f7fc013f650 2024-03-29T11:36:01.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fc5675640 10 monclient: get_auth_request con 0x7f7fa406dc60 auth_method 0 2024-03-29T11:36:01.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f7fbc084b50 con 0x7f7fc013f650 2024-03-29T11:36:01.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fb5ffb640 10 client.4283.objecter ms_dispatch 0x7f7fc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:01.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fb5ffb640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:01.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fb5ffb640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:01.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fb5ffb640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T11:36:01.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.408+0000 7f7fb5ffb640 10 client.4283.objecter ms_handle_connect 0x7f7fa406dc60 2024-03-29T11:36:01.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.417+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fbc0504b0 con 0x7f7fc013f650 2024-03-29T11:36:01.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.417+0000 7f7fb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:01.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.417+0000 7f7fb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.761+0000 7f7fc78d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.761+0000 7f7fc78d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.761+0000 7f7fc78d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.761+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fa0005330 con 0x7f7fc013f650 2024-03-29T11:36:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.762+0000 7f7fb5ffb640 1 -- 172.21.15.22:0/1858311991 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f7fbc0523e0 con 0x7f7fc013f650 2024-03-29T11:36:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.762+0000 7f7fb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.762+0000 7f7fb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:01.762 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:36:01.762 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.765+0000 7f7fc78d9640 10 librados: watch_flush enter 2024-03-29T11:36:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.765+0000 7f7fc78d9640 10 librados: watch_flush exit 2024-03-29T11:36:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.765+0000 7f7fc78d9640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T11:36:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.765+0000 7f7fc78d9640 10 client.4283.objecter successfully canceled tick 2024-03-29T11:36:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.766+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 >> v1:172.21.15.204:6816/3848099522 conn(0x7f7fa406dc60 legacy=0x7f7fa4070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.766+0000 7f7fc78d9640 10 monclient: shutdown 2024-03-29T11:36:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.766+0000 7f7fc78d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.766+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 >> v1:172.21.15.22:6790/0 conn(0x7f7fc013f650 legacy=0x7f7fc01db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.766+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 shutdown_connections 2024-03-29T11:36:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.766+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 >> 172.21.15.22:0/1858311991 conn(0x7f7fc0066a00 msgr2=0x7f7fc00b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.767+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 shutdown_connections 2024-03-29T11:36:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.767+0000 7f7fc78d9640 1 -- 172.21.15.22:0/1858311991 wait complete. 2024-03-29T11:36:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:01.767+0000 7f7fc78d9640 1 librados: shutdown 2024-03-29T11:36:01.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:08.589 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:08.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.738+0000 7fc3dc856640 10 monclient: get_monmap_and_config 2024-03-29T11:36:08.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.738+0000 7fc3dc856640 10 monclient: build_initial_monmap 2024-03-29T11:36:08.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.738+0000 7fc3dc856640 10 monclient: monmap: 2024-03-29T11:36:08.738 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:08.738 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:08.738 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:36:08.740106+0000 2024-03-29T11:36:08.738 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:36:08.740106+0000 2024-03-29T11:36:08.739 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:08.739 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:08.739 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:08.739 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:36:08.739 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:36:08.739 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:08.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.740+0000 7fc3dc856640 1 Processor -- start 2024-03-29T11:36:08.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.740+0000 7fc3dc856640 1 -- start start 2024-03-29T11:36:08.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.740+0000 7fc3dc856640 10 monclient: init 2024-03-29T11:36:08.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.741+0000 7fc3dc856640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:08.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.741+0000 7fc3dc856640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.743+0000 7fc3dc856640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d814a400 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.743+0000 7fc3dc856640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d80aa500 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.743+0000 7fc3dc856640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d8149800 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.743+0000 7fc3dc856640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d80aca50 con 0x7fc3d80aa500 2024-03-29T11:36:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.743+0000 7fc3dc856640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d8149cb0 con 0x7fc3d814a400 2024-03-29T11:36:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.743+0000 7fc3dc856640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d80acc80 con 0x7fc3d8149800 2024-03-29T11:36:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.743+0000 7fc3dc856640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.743+0000 7fc3dc856640 10 monclient(hunting): authenticate will time out at 1019.020747s 2024-03-29T11:36:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.744+0000 7fc3d6d9d640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc3d814a400 0x7fc3d80aac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:35537/0 (socket says 172.21.15.22:35537) 2024-03-29T11:36:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.744+0000 7fc3d6d9d640 1 -- 172.21.15.22:0/188325597 learned_addr learned my addr 172.21.15.22:0/188325597 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.745+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655434948 0 0) 0x7fc3d8149cb0 con 0x7fc3d814a400 2024-03-29T11:36:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.745+0000 7fc3d5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:36:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.745+0000 7fc3d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.745+0000 7fc3d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.745+0000 7fc3d5d9b640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:36:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.745+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c0003520 con 0x7fc3d814a400 2024-03-29T11:36:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.745+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2858409520 0 0) 0x7fc3c0003520 con 0x7fc3d814a400 2024-03-29T11:36:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.746+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 >> v1:172.21.15.22:6790/0 conn(0x7fc3d8149800 legacy=0x7fc3d8151ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:08.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 >> v1:172.21.15.22:6789/0 conn(0x7fc3d80aa500 legacy=0x7fc3d80ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:36:08.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 1 monclient: found mon.noname-c 2024-03-29T11:36:08.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:08.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:36:08.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d80ac350 con 0x7fc3d814a400 2024-03-29T11:36:08.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:36:08.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc3c8001690 con 0x7fc3d814a400 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3d5d9b640 10 monclient: dump: 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:08.748 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3dc856640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.747+0000 7fc3dc856640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.748+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3c8001860 con 0x7fc3d814a400 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.748+0000 7fc3d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.748+0000 7fc3d5d9b640 1 -- 172.21.15.22:0/188325597 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc3c8001cb0 con 0x7fc3d814a400 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.748+0000 7fc3d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.748+0000 7fc3d5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.748+0000 7fc3d5d9b640 10 monclient: dump: 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:08.749 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.750 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:08.750 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:08.750 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:08.750 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:08.750 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:08.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.748+0000 7fc3dc856640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:08.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.748+0000 7fc3dc856640 4 set_mon_vals no callback set 2024-03-29T11:36:08.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.749+0000 7fc3dc856640 10 monclient: shutdown 2024-03-29T11:36:08.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.749+0000 7fc3dc856640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:08.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.749+0000 7fc3dc856640 1 -- 172.21.15.22:0/188325597 >> v1:172.21.15.204:6789/0 conn(0x7fc3d814a400 legacy=0x7fc3d80aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.750+0000 7fc3dc856640 1 -- 172.21.15.22:0/188325597 shutdown_connections 2024-03-29T11:36:08.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.750+0000 7fc3dc856640 1 -- 172.21.15.22:0/188325597 >> 172.21.15.22:0/188325597 conn(0x7fc3d80b3a30 msgr2=0x7fc3d80b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.750+0000 7fc3dc856640 1 -- 172.21.15.22:0/188325597 shutdown_connections 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.750+0000 7fc3dc856640 1 -- 172.21.15.22:0/188325597 wait complete. 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.752+0000 7fc3dc856640 10 monclient: build_initial_monmap 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.752+0000 7fc3dc856640 10 monclient: monmap: 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:36:08.752 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:08.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.752+0000 7fc3dc856640 1 librados: starting msgr at 2024-03-29T11:36:08.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.752+0000 7fc3dc856640 1 librados: starting objecter 2024-03-29T11:36:08.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.752+0000 7fc3dc856640 1 Processor -- start 2024-03-29T11:36:08.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.753+0000 7fc3dc856640 1 -- start start 2024-03-29T11:36:08.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.753+0000 7fc3dc856640 1 librados: setting wanted keys 2024-03-29T11:36:08.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.753+0000 7fc3dc856640 1 librados: calling monclient init 2024-03-29T11:36:08.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.753+0000 7fc3dc856640 10 monclient: init 2024-03-29T11:36:08.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.753+0000 7fc3dc856640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:08.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.753+0000 7fc3dc856640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:08.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.756+0000 7fc3dc856640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d80aa500 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:08.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.756+0000 7fc3d6d9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc3d80aa500 0x7fc3d81dee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40001/0 (socket says 172.21.15.22:40001) 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.758+0000 7fc3dc856640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d8149800 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.758+0000 7fc3dc856640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d814a400 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.758+0000 7fc3dc856640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d81de9e0 con 0x7fc3d814a400 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.758+0000 7fc3dc856640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d8148b20 con 0x7fc3d8149800 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.758+0000 7fc3dc856640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d813efd0 con 0x7fc3d80aa500 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.758+0000 7fc3dc856640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.758+0000 7fc3dc856640 10 monclient(hunting): authenticate will time out at 1019.035816s 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.759+0000 7fc3d6d9d640 1 -- 172.21.15.22:0/3480491985 learned_addr learned my addr 172.21.15.22:0/3480491985 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 client.?.objecter ms_handle_connect 0x7fc3d80aa500 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832706365 0 0) 0x7fc3d813efd0 con 0x7fc3d80aa500 2024-03-29T11:36:08.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:08.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:08.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:08.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:36:08.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c4004eb0 con 0x7fc3d80aa500 2024-03-29T11:36:08.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 27723130 0 0) 0x7fc3c4004eb0 con 0x7fc3d80aa500 2024-03-29T11:36:08.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3d813efd0 con 0x7fc3d80aa500 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc3c8001c30 con 0x7fc3d80aa500 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 monclient(hunting): dump: 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:08.763 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.760+0000 7fc3bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.761+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3886445868 0 0) 0x7fc3d813efd0 con 0x7fc3d80aa500 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.761+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 >> v1:172.21.15.204:6789/0 conn(0x7fc3d8149800 legacy=0x7fc3d81d8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.761+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 >> v1:172.21.15.22:6789/0 conn(0x7fc3d814a400 legacy=0x7fc3d81e5d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3bf7fe640 1 monclient: found mon.c 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3bf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d81df580 con 0x7fc3d80aa500 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:36:08.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 10 monclient: _renew_subs 2024-03-29T11:36:08.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:08.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d81df830 con 0x7fc3d80aa500 2024-03-29T11:36:08.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:08.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 10 monclient: _renew_subs 2024-03-29T11:36:08.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:08.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d81d8e30 con 0x7fc3d80aa500 2024-03-29T11:36:08.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.763+0000 7fc3dc856640 1 librados: init done 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.764+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3c8004a90 con 0x7fc3d80aa500 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.764+0000 7fc3bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.764+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc3c8004ee0 con 0x7fc3d80aa500 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.764+0000 7fc3bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.764+0000 7fc3bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.764+0000 7fc3bf7fe640 10 monclient: dump: 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:08.767 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.764+0000 7fc3d559a640 4 set_mon_vals no callback set 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.764+0000 7fc3dc856640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.765+0000 7fc3dc856640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.766+0000 7fc3dc856640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.766+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d804f730 con 0x7fc3d80aa500 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.767+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fc3c8005040 con 0x7fc3d80aa500 2024-03-29T11:36:08.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.767+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fc3c8084c20 con 0x7fc3d80aa500 2024-03-29T11:36:08.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.767+0000 7fc3bf7fe640 10 client.4286.objecter ms_dispatch 0x7fc3d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:08.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.767+0000 7fc3bf7fe640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:08.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.767+0000 7fc3bf7fe640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:08.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.767+0000 7fc3bf7fe640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T11:36:08.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.769+0000 7fc3d659c640 10 monclient: get_auth_request con 0x7fc3c406dd50 auth_method 0 2024-03-29T11:36:08.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.770+0000 7fc3bf7fe640 10 client.4286.objecter ms_handle_connect 0x7fc3c406dd50 2024-03-29T11:36:08.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.775+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3c8050500 con 0x7fc3d80aa500 2024-03-29T11:36:08.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.775+0000 7fc3bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:08.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:08.775+0000 7fc3bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:09.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.112+0000 7fc3dc856640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.112+0000 7fc3dc856640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.112+0000 7fc3dc856640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:09.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.112+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3d80b28d0 con 0x7fc3d80aa500 2024-03-29T11:36:09.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.114+0000 7fc3bf7fe640 1 -- 172.21.15.22:0/3480491985 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fc3c8052430 con 0x7fc3d80aa500 2024-03-29T11:36:09.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.114+0000 7fc3bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.114+0000 7fc3bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:09.115 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:36:09.115 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.120+0000 7fc3dc856640 10 librados: watch_flush enter 2024-03-29T11:36:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.120+0000 7fc3dc856640 10 librados: watch_flush exit 2024-03-29T11:36:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.120+0000 7fc3dc856640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T11:36:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.120+0000 7fc3dc856640 10 client.4286.objecter successfully canceled tick 2024-03-29T11:36:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.120+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 >> v1:172.21.15.204:6816/3848099522 conn(0x7fc3c406dd50 legacy=0x7fc3c4070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.120+0000 7fc3dc856640 10 monclient: shutdown 2024-03-29T11:36:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.120+0000 7fc3dc856640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.120+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 >> v1:172.21.15.22:6790/0 conn(0x7fc3d80aa500 legacy=0x7fc3d81dee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.123+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 shutdown_connections 2024-03-29T11:36:09.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.123+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 >> 172.21.15.22:0/3480491985 conn(0x7fc3d80b3a30 msgr2=0x7fc3d80b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:09.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.123+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 shutdown_connections 2024-03-29T11:36:09.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.123+0000 7fc3dc856640 1 -- 172.21.15.22:0/3480491985 wait complete. 2024-03-29T11:36:09.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:09.123+0000 7fc3dc856640 1 librados: shutdown 2024-03-29T11:36:09.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:15.948 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:16.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.099+0000 7f6ce2614640 10 monclient: get_monmap_and_config 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.099+0000 7f6ce2614640 10 monclient: build_initial_monmap 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.099+0000 7f6ce2614640 10 monclient: monmap: 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:36:16.100994+0000 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:36:16.100994+0000 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:36:16.099 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:36:16.100 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:16.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.100+0000 7f6ce2614640 1 Processor -- start 2024-03-29T11:36:16.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.103+0000 7f6ce2614640 1 -- start start 2024-03-29T11:36:16.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.103+0000 7f6ce2614640 10 monclient: init 2024-03-29T11:36:16.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 10 monclient(hunting): picked mon.noname-b con 0x7f6cdc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 10 monclient(hunting): picked mon.noname-c con 0x7f6cdc13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 10 monclient(hunting): picked mon.noname-a con 0x7f6cdc0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0aaef0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc145bd0 con 0x7f6cdc13f650 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc145eb0 con 0x7f6cdc1457f0 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:16.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.104+0000 7f6ce2614640 10 monclient(hunting): authenticate will time out at 1026.381848s 2024-03-29T11:36:16.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.108+0000 7f6ce0bb1640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f6cdc0a9d70 0x7f6cdc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36111/0 (socket says 172.21.15.22:36111) 2024-03-29T11:36:16.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.108+0000 7f6ce0bb1640 1 -- 172.21.15.22:0/2134122443 learned_addr learned my addr 172.21.15.22:0/2134122443 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.111+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238882044 0 0) 0x7f6cdc0aaef0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.111+0000 7f6cdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.111+0000 7f6cdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.111+0000 7f6cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.111+0000 7f6cdaffd640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:36:16.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.111+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc8003520 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.112+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711007605 0 0) 0x7f6cdc145eb0 con 0x7f6cdc1457f0 2024-03-29T11:36:16.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.112+0000 7f6cdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.112+0000 7f6cdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.112+0000 7f6cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.113+0000 7f6cdaffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:36:16.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.113+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc0aaef0 con 0x7f6cdc1457f0 2024-03-29T11:36:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.113+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3927424187 0 0) 0x7f6cc8003520 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 >> v1:172.21.15.22:6790/0 conn(0x7f6cdc1457f0 legacy=0x7f6cdc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 >> v1:172.21.15.204:6789/0 conn(0x7f6cdc13f650 legacy=0x7f6cdc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 1 monclient: found mon.noname-a 2024-03-29T11:36:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:36:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc146340 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f6cc4001690 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6cdaffd640 10 monclient: dump: 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:16.117 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:16.118 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:16.118 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:16.118 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:16.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6ce2614640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:36:16.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.116+0000 7f6ce2614640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:16.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.120+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cc40017f0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.120+0000 7f6cdaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:16.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.120+0000 7f6cdaffd640 1 -- 172.21.15.22:0/2134122443 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f6cc4001c40 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.120+0000 7f6cdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.120+0000 7f6cdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:16.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.120+0000 7f6cdaffd640 10 monclient: dump: 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.120+0000 7f6ce2614640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.120+0000 7f6ce2614640 4 set_mon_vals no callback set 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.121+0000 7f6ce2614640 10 monclient: shutdown 2024-03-29T11:36:16.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.121+0000 7f6ce2614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.121+0000 7f6ce2614640 1 -- 172.21.15.22:0/2134122443 >> v1:172.21.15.22:6789/0 conn(0x7f6cdc0a9d70 legacy=0x7f6cdc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.121+0000 7f6ce2614640 1 -- 172.21.15.22:0/2134122443 shutdown_connections 2024-03-29T11:36:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.121+0000 7f6ce2614640 1 -- 172.21.15.22:0/2134122443 >> 172.21.15.22:0/2134122443 conn(0x7f6cdc066a00 msgr2=0x7f6cdc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.121+0000 7f6ce2614640 1 -- 172.21.15.22:0/2134122443 shutdown_connections 2024-03-29T11:36:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.122+0000 7f6ce2614640 1 -- 172.21.15.22:0/2134122443 wait complete. 2024-03-29T11:36:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.123+0000 7f6ce2614640 10 monclient: build_initial_monmap 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.123+0000 7f6ce2614640 10 monclient: monmap: 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.124+0000 7f6ce2614640 1 librados: starting msgr at 2024-03-29T11:36:16.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.124+0000 7f6ce2614640 1 librados: starting objecter 2024-03-29T11:36:16.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.124+0000 7f6ce2614640 1 Processor -- start 2024-03-29T11:36:16.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.124+0000 7f6ce2614640 1 -- start start 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.124+0000 7f6ce2614640 1 librados: setting wanted keys 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.124+0000 7f6ce2614640 1 librados: calling monclient init 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.124+0000 7f6ce2614640 10 monclient: init 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 10 monclient(hunting): picked mon.noname-a con 0x7f6cdc0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 10 monclient(hunting): picked mon.noname-b con 0x7f6cdc13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 10 monclient(hunting): picked mon.noname-c con 0x7f6cdc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc135870 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc141ec0 con 0x7f6cdc13f650 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc13b850 con 0x7f6cdc1457f0 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.125+0000 7f6ce2614640 10 monclient(hunting): authenticate will time out at 1026.402226s 2024-03-29T11:36:16.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.127+0000 7f6cdbfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f6cdc0a9d70 0x7f6cdc1d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36239/0 (socket says 172.21.15.22:36239) 2024-03-29T11:36:16.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.127+0000 7f6cdbfff640 1 -- 172.21.15.22:0/3512114570 learned_addr learned my addr 172.21.15.22:0/3512114570 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:16.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.128+0000 7f6cd8ff9640 10 client.?.objecter ms_handle_connect 0x7f6cdc0a9d70 2024-03-29T11:36:16.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.128+0000 7f6cd8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.128+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095043375 0 0) 0x7f6cdc135870 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.128+0000 7f6cd8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:36:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cbc004eb0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 client.?.objecter ms_handle_connect 0x7f6cdc13f650 2024-03-29T11:36:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049375728 0 0) 0x7f6cdc141ec0 con 0x7f6cdc13f650 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc135870 con 0x7f6cdc13f650 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.131+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838370166 0 0) 0x7f6cbc004eb0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cdc141ec0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f6cd4001990 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:16.135 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174983540 0 0) 0x7f6cdc135870 con 0x7f6cdc13f650 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cbc004eb0 con 0x7f6cdc13f650 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1597846778 0 0) 0x7f6cdc141ec0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.132+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 >> v1:172.21.15.22:6790/0 conn(0x7f6cdc1457f0 legacy=0x7f6cdc135160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 >> v1:172.21.15.204:6789/0 conn(0x7f6cdc13f650 legacy=0x7f6cdc134a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 1 monclient: found mon.a 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc140f80 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6cd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 10 monclient: _renew_subs 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc1d7cf0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 10 monclient: _renew_subs 2024-03-29T11:36:16.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc142b10 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.133+0000 7f6ce2614640 1 librados: init done 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.134+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cd4001af0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.134+0000 7f6cd8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.134+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f6cd40038a0 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.134+0000 7f6cd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.134+0000 7f6cd8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.134+0000 7f6cd8ff9640 10 monclient: dump: 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:16.138 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.139 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:16.139 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:16.139 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:16.139 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:16.139 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:16.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.134+0000 7f6cd9ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:16.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.136+0000 7f6ce2614640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.136+0000 7f6ce2614640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.136+0000 7f6ce2614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:16.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.136+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc0005080 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.136+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f6cd4003a00 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.137+0000 7f6cdb7fe640 10 monclient: get_auth_request con 0x7f6cbc06dde0 auth_method 0 2024-03-29T11:36:16.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.137+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f6cd4084980 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.137+0000 7f6cd8ff9640 10 client.4293.objecter ms_dispatch 0x7f6cdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:16.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.138+0000 7f6cd8ff9640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:16.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.138+0000 7f6cd8ff9640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:16.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.138+0000 7f6cd8ff9640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T11:36:16.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.138+0000 7f6cd8ff9640 10 client.4293.objecter ms_handle_connect 0x7f6cbc06dde0 2024-03-29T11:36:16.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.149+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6cd4050260 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.149+0000 7f6cd8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.149+0000 7f6cd8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:16.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.472+0000 7f6ce2614640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:16.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.473+0000 7f6ce2614640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:16.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.473+0000 7f6ce2614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:16.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.473+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cc0005330 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.475 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:36:16.475 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:16.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.473+0000 7f6cd8ff9640 1 -- 172.21.15.22:0/3512114570 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f6cd4052190 con 0x7f6cdc0a9d70 2024-03-29T11:36:16.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.473+0000 7f6cd8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:16.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.473+0000 7f6cd8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:16.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.477+0000 7f6ce2614640 10 librados: watch_flush enter 2024-03-29T11:36:16.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.477+0000 7f6ce2614640 10 librados: watch_flush exit 2024-03-29T11:36:16.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.477+0000 7f6ce2614640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T11:36:16.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.477+0000 7f6ce2614640 10 client.4293.objecter successfully canceled tick 2024-03-29T11:36:16.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.477+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 >> v1:172.21.15.204:6816/3848099522 conn(0x7f6cbc06dde0 legacy=0x7f6cbc0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.477+0000 7f6ce2614640 10 monclient: shutdown 2024-03-29T11:36:16.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.477+0000 7f6ce2614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:16.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.477+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 >> v1:172.21.15.22:6789/0 conn(0x7f6cdc0a9d70 legacy=0x7f6cdc1d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.478+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 shutdown_connections 2024-03-29T11:36:16.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.478+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 >> 172.21.15.22:0/3512114570 conn(0x7f6cdc066a00 msgr2=0x7f6cdc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.479+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 shutdown_connections 2024-03-29T11:36:16.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.479+0000 7f6ce2614640 1 -- 172.21.15.22:0/3512114570 wait complete. 2024-03-29T11:36:16.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:16.483+0000 7f6ce2614640 1 librados: shutdown 2024-03-29T11:36:16.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:23.308 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:23.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.455+0000 7fed33fff640 10 monclient: get_monmap_and_config 2024-03-29T11:36:23.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.455+0000 7fed33fff640 10 monclient: build_initial_monmap 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.455+0000 7fed33fff640 10 monclient: monmap: 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:36:23.456584+0000 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:36:23.456584+0000 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:36:23.455 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:23.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.458+0000 7fed33fff640 1 Processor -- start 2024-03-29T11:36:23.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.458+0000 7fed33fff640 1 -- start start 2024-03-29T11:36:23.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.458+0000 7fed33fff640 10 monclient: init 2024-03-29T11:36:23.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.459+0000 7fed33fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:23.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.459+0000 7fed33fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:23.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.463+0000 7fed33fff640 10 monclient(hunting): picked mon.noname-a con 0x7fed2c038c60 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:23.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.463+0000 7fed33fff640 10 monclient(hunting): picked mon.noname-b con 0x7fed2c0324b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:23.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.467+0000 7fed32ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fed2c038c60 0x7fed2c0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55155/0 (socket says 172.21.15.22:55155) 2024-03-29T11:36:23.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.467+0000 7fed32ffd640 1 -- 172.21.15.22:0/2886647872 learned_addr learned my addr 172.21.15.22:0/2886647872 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:23.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.467+0000 7fed33fff640 10 monclient(hunting): picked mon.noname-c con 0x7fed2c0d11e0 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:23.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.467+0000 7fed33fff640 1 -- 172.21.15.22:0/2886647872 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c0caf80 con 0x7fed2c038c60 2024-03-29T11:36:23.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.467+0000 7fed33fff640 1 -- 172.21.15.22:0/2886647872 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c039850 con 0x7fed2c0d11e0 2024-03-29T11:36:23.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.467+0000 7fed33fff640 1 -- 172.21.15.22:0/2886647872 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c0c78d0 con 0x7fed2c0324b0 2024-03-29T11:36:23.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.467+0000 7fed33fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:23.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.467+0000 7fed33fff640 10 monclient(hunting): authenticate will time out at 1033.744725s 2024-03-29T11:36:23.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429429093 0 0) 0x7fed2c0caf80 con 0x7fed2c038c60 2024-03-29T11:36:23.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:36:23.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c003520 con 0x7fed2c038c60 2024-03-29T11:36:23.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 144797768 0 0) 0x7fed1c003520 con 0x7fed2c038c60 2024-03-29T11:36:23.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 >> v1:172.21.15.22:6790/0 conn(0x7fed2c0324b0 legacy=0x7fed2c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:23.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 >> v1:172.21.15.204:6789/0 conn(0x7fed2c0d11e0 legacy=0x7fed2c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:23.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 monclient: found mon.noname-a 2024-03-29T11:36:23.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:23.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:36:23.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c0c7a60 con 0x7fed2c038c60 2024-03-29T11:36:23.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fed24001690 con 0x7fed2c038c60 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: dump: 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:23.476 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed24001860 con 0x7fed2c038c60 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 1 -- 172.21.15.22:0/2886647872 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fed24001cb0 con 0x7fed2c038c60 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.470+0000 7fed31ffb640 10 monclient: dump: 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:23.477 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.473+0000 7fed33fff640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.473+0000 7fed33fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.473+0000 7fed33fff640 4 set_mon_vals no callback set 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.474+0000 7fed33fff640 10 monclient: shutdown 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.474+0000 7fed33fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:23.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.474+0000 7fed33fff640 1 -- 172.21.15.22:0/2886647872 >> v1:172.21.15.22:6789/0 conn(0x7fed2c038c60 legacy=0x7fed2c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.476+0000 7fed33fff640 1 -- 172.21.15.22:0/2886647872 shutdown_connections 2024-03-29T11:36:23.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.476+0000 7fed33fff640 1 -- 172.21.15.22:0/2886647872 >> 172.21.15.22:0/2886647872 conn(0x7fed2c005c20 msgr2=0x7fed2c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:23.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.479+0000 7fed33fff640 1 -- 172.21.15.22:0/2886647872 shutdown_connections 2024-03-29T11:36:23.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.481+0000 7fed33fff640 1 -- 172.21.15.22:0/2886647872 wait complete. 2024-03-29T11:36:23.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.482+0000 7fed33fff640 10 monclient: build_initial_monmap 2024-03-29T11:36:23.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.482+0000 7fed33fff640 10 monclient: monmap: 2024-03-29T11:36:23.481 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:23.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.483+0000 7fed33fff640 1 librados: starting msgr at 2024-03-29T11:36:23.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.484+0000 7fed33fff640 1 librados: starting objecter 2024-03-29T11:36:23.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.485+0000 7fed33fff640 1 Processor -- start 2024-03-29T11:36:23.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.485+0000 7fed33fff640 1 -- start start 2024-03-29T11:36:23.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.485+0000 7fed33fff640 1 librados: setting wanted keys 2024-03-29T11:36:23.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.485+0000 7fed33fff640 1 librados: calling monclient init 2024-03-29T11:36:23.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.485+0000 7fed33fff640 10 monclient: init 2024-03-29T11:36:23.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:23.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:23.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 10 monclient(hunting): picked mon.noname-b con 0x7fed2c0324b0 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:23.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 10 monclient(hunting): picked mon.noname-c con 0x7fed2c038c60 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:23.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 10 monclient(hunting): picked mon.noname-a con 0x7fed2c0d11e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:23.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c036430 con 0x7fed2c0d11e0 2024-03-29T11:36:23.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c02f9d0 con 0x7fed2c0324b0 2024-03-29T11:36:23.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c0babb0 con 0x7fed2c038c60 2024-03-29T11:36:23.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:23.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.486+0000 7fed33fff640 10 monclient(hunting): authenticate will time out at 1033.763828s 2024-03-29T11:36:23.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.490+0000 7fed327fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fed2c038c60 0x7fed2c035d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45327/0 (socket says 172.21.15.22:45327) 2024-03-29T11:36:23.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.490+0000 7fed327fc640 1 -- 172.21.15.22:0/4188782062 learned_addr learned my addr 172.21.15.22:0/4188782062 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:23.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 10 client.?.objecter ms_handle_connect 0x7fed2c038c60 2024-03-29T11:36:23.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:23.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 10 client.?.objecter ms_handle_connect 0x7fed2c0d11e0 2024-03-29T11:36:23.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:23.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737098010 0 0) 0x7fed2c036430 con 0x7fed2c0d11e0 2024-03-29T11:36:23.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:36:23.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.491+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed20004eb0 con 0x7fed2c0d11e0 2024-03-29T11:36:23.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 client.?.objecter ms_handle_connect 0x7fed2c0324b0 2024-03-29T11:36:23.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:23.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 760122873 0 0) 0x7fed20004eb0 con 0x7fed2c0d11e0 2024-03-29T11:36:23.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed2c036430 con 0x7fed2c0d11e0 2024-03-29T11:36:23.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504997009 0 0) 0x7fed2c0babb0 con 0x7fed2c038c60 2024-03-29T11:36:23.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:36:23.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed20004eb0 con 0x7fed2c038c60 2024-03-29T11:36:23.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fed3404fdd0 con 0x7fed2c0d11e0 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): dump: 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:23.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181049486 0 0) 0x7fed2c02f9d0 con 0x7fed2c0324b0 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2c0babb0 con 0x7fed2c0324b0 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2629893389 0 0) 0x7fed2c036430 con 0x7fed2c0d11e0 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 >> v1:172.21.15.22:6790/0 conn(0x7fed2c038c60 legacy=0x7fed2c035d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 >> v1:172.21.15.204:6789/0 conn(0x7fed2c0324b0 legacy=0x7fed2c02dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.494+0000 7fed1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed1b7fe640 1 monclient: found mon.a 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed1b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:23.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c02d600 con 0x7fed2c0d11e0 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 10 monclient: _renew_subs 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2c02d300 con 0x7fed2c0d11e0 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 10 monclient: _renew_subs 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2c0097b0 con 0x7fed2c0d11e0 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.495+0000 7fed33fff640 1 librados: init done 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.496+0000 7fed33fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.496+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed3404f400 con 0x7fed2c0d11e0 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.496+0000 7fed33fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.496+0000 7fed33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.496+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2c007b90 con 0x7fed2c0d11e0 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.497+0000 7fed1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.497+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fed34075f10 con 0x7fed2c0d11e0 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.498+0000 7fed1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.498+0000 7fed1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.499+0000 7fed1b7fe640 10 monclient: dump: 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:23.505 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:23.506 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:23.506 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:23.506 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:23.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.499+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fed34067bf0 con 0x7fed2c0d11e0 2024-03-29T11:36:23.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.499+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fed340fe0e0 con 0x7fed2c0d11e0 2024-03-29T11:36:23.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.499+0000 7fed1b7fe640 10 client.4299.objecter ms_dispatch 0x7fed34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:23.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.499+0000 7fed1b7fe640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:23.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.499+0000 7fed1b7fe640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:23.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.500+0000 7fed1b7fe640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T11:36:23.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.500+0000 7fed32ffd640 10 monclient: get_auth_request con 0x7fed2006dbe0 auth_method 0 2024-03-29T11:36:23.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.500+0000 7fed30ff9640 4 set_mon_vals no callback set 2024-03-29T11:36:23.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.500+0000 7fed1b7fe640 10 client.4299.objecter ms_handle_connect 0x7fed2006dbe0 2024-03-29T11:36:23.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.505+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed340c9a40 con 0x7fed2c0d11e0 2024-03-29T11:36:23.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.505+0000 7fed1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.505+0000 7fed1b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:23.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.839+0000 7fed33fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.839+0000 7fed33fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.839+0000 7fed33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.839+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed2c004cd0 con 0x7fed2c0d11e0 2024-03-29T11:36:23.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.843+0000 7fed1b7fe640 1 -- 172.21.15.22:0/4188782062 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fed340cb970 con 0x7fed2c0d11e0 2024-03-29T11:36:23.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.843+0000 7fed1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:23.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.843+0000 7fed1b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:23.845 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:36:23.845 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:23.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.846+0000 7fed33fff640 10 librados: watch_flush enter 2024-03-29T11:36:23.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.846+0000 7fed33fff640 10 librados: watch_flush exit 2024-03-29T11:36:23.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.846+0000 7fed33fff640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T11:36:23.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.846+0000 7fed33fff640 10 client.4299.objecter successfully canceled tick 2024-03-29T11:36:23.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.847+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 >> v1:172.21.15.204:6816/3848099522 conn(0x7fed2006dbe0 legacy=0x7fed200700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.847+0000 7fed33fff640 10 monclient: shutdown 2024-03-29T11:36:23.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.847+0000 7fed33fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:23.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.847+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 >> v1:172.21.15.22:6789/0 conn(0x7fed2c0d11e0 legacy=0x7fed2c163370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.851+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 shutdown_connections 2024-03-29T11:36:23.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.851+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 >> 172.21.15.22:0/4188782062 conn(0x7fed2c005c20 msgr2=0x7fed2c031bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:23.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.852+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 shutdown_connections 2024-03-29T11:36:23.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.852+0000 7fed33fff640 1 -- 172.21.15.22:0/4188782062 wait complete. 2024-03-29T11:36:23.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:23.852+0000 7fed33fff640 1 librados: shutdown 2024-03-29T11:36:23.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:30.675 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:30.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.827+0000 7f5cb9477640 10 monclient: get_monmap_and_config 2024-03-29T11:36:30.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.827+0000 7f5cb9477640 10 monclient: build_initial_monmap 2024-03-29T11:36:30.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.827+0000 7f5cb9477640 10 monclient: monmap: 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:36:30.828600+0000 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:36:30.828600+0000 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:36:30.827 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:30.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.831+0000 7f5cb9477640 1 Processor -- start 2024-03-29T11:36:30.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.831+0000 7f5cb9477640 1 -- start start 2024-03-29T11:36:30.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.831+0000 7f5cb9477640 10 monclient: init 2024-03-29T11:36:30.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 10 monclient(hunting): picked mon.noname-a con 0x7f5cac02b4c0 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 10 monclient(hunting): picked mon.noname-c con 0x7f5cac00b8a0 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 10 monclient(hunting): picked mon.noname-b con 0x7f5cac02a5a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac02c5c0 con 0x7f5cac02b4c0 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac0c11a0 con 0x7f5cac00b8a0 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac02c830 con 0x7f5cac02a5a0 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:30.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.832+0000 7f5cb9477640 10 monclient(hunting): authenticate will time out at 1041.109576s 2024-03-29T11:36:30.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.835+0000 7f5cb3fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f5cac02b4c0 0x7f5cac006890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36587/0 (socket says 172.21.15.22:36587) 2024-03-29T11:36:30.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.835+0000 7f5cb3fff640 1 -- 172.21.15.22:0/2886450936 learned_addr learned my addr 172.21.15.22:0/2886450936 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:30.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620545449 0 0) 0x7f5cac0c11a0 con 0x7f5cac00b8a0 2024-03-29T11:36:30.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:30.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ca4003520 con 0x7f5cac00b8a0 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1441292855 0 0) 0x7f5ca4003520 con 0x7f5cac00b8a0 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 >> v1:172.21.15.22:6790/0 conn(0x7f5cac02a5a0 legacy=0x7f5cac0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 >> v1:172.21.15.22:6789/0 conn(0x7f5cac02b4c0 legacy=0x7f5cac006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac02afa0 con 0x7f5cac00b8a0 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:30.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5cb404f730 con 0x7f5cac00b8a0 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: dump: 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:30.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cb40008d0 con 0x7f5cac00b8a0 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 1 -- 172.21.15.22:0/2886450936 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5cb404f730 con 0x7f5cac00b8a0 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.837+0000 7f5cb2ffd640 10 monclient: dump: 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:30.842 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.839+0000 7f5cb9477640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.839+0000 7f5cb9477640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.839+0000 7f5cb9477640 4 set_mon_vals no callback set 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.840+0000 7f5cb9477640 10 monclient: shutdown 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.840+0000 7f5cb9477640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.840+0000 7f5cb9477640 1 -- 172.21.15.22:0/2886450936 >> v1:172.21.15.204:6789/0 conn(0x7f5cac00b8a0 legacy=0x7f5cac0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.840+0000 7f5cb9477640 1 -- 172.21.15.22:0/2886450936 shutdown_connections 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.840+0000 7f5cb9477640 1 -- 172.21.15.22:0/2886450936 >> 172.21.15.22:0/2886450936 conn(0x7f5cac0058f0 msgr2=0x7f5cac005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:30.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.841+0000 7f5cb9477640 1 -- 172.21.15.22:0/2886450936 shutdown_connections 2024-03-29T11:36:30.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.843+0000 7f5cb9477640 1 -- 172.21.15.22:0/2886450936 wait complete. 2024-03-29T11:36:30.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.844+0000 7f5cb9477640 10 monclient: build_initial_monmap 2024-03-29T11:36:30.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.844+0000 7f5cb9477640 10 monclient: monmap: 2024-03-29T11:36:30.844 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:30.844 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:30.844 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:36:30.844 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.844+0000 7f5cb9477640 1 librados: starting msgr at 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.844+0000 7f5cb9477640 1 librados: starting objecter 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.844+0000 7f5cb9477640 1 Processor -- start 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.845+0000 7f5cb9477640 1 -- start start 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.845+0000 7f5cb9477640 1 librados: setting wanted keys 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.845+0000 7f5cb9477640 1 librados: calling monclient init 2024-03-29T11:36:30.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.845+0000 7f5cb9477640 10 monclient: init 2024-03-29T11:36:30.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:30.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:30.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 10 monclient(hunting): picked mon.noname-c con 0x7f5cac00b8a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:30.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 10 monclient(hunting): picked mon.noname-b con 0x7f5cac02a5a0 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:30.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 10 monclient(hunting): picked mon.noname-a con 0x7f5cac02b4c0 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:30.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac0ca490 con 0x7f5cac02b4c0 2024-03-29T11:36:30.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac0c9f70 con 0x7f5cac02a5a0 2024-03-29T11:36:30.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cac0c4880 con 0x7f5cac00b8a0 2024-03-29T11:36:30.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:30.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.846+0000 7f5cb9477640 10 monclient(hunting): authenticate will time out at 1041.123156s 2024-03-29T11:36:30.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.849+0000 7f5cb8c76640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f5cac02b4c0 0x7f5cac17a660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36759/0 (socket says 172.21.15.22:36759) 2024-03-29T11:36:30.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.849+0000 7f5cb8c76640 1 -- 172.21.15.22:0/2115753699 learned_addr learned my addr 172.21.15.22:0/2115753699 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:30.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.850+0000 7f5cb0ff9640 10 client.?.objecter ms_handle_connect 0x7f5cac02b4c0 2024-03-29T11:36:30.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.850+0000 7f5cb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:30.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.850+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353212101 0 0) 0x7f5cac0ca490 con 0x7f5cac02b4c0 2024-03-29T11:36:30.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.850+0000 7f5cb0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:30.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.850+0000 7f5cb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:30.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.850+0000 7f5cb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:30.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.850+0000 7f5cb0ff9640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:36:30.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.850+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c98004eb0 con 0x7f5cac02b4c0 2024-03-29T11:36:30.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1339755998 0 0) 0x7f5c98004eb0 con 0x7f5cac02b4c0 2024-03-29T11:36:30.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cac0ca490 con 0x7f5cac02b4c0 2024-03-29T11:36:30.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5ca8001980 con 0x7f5cac02b4c0 2024-03-29T11:36:30.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:30.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:30.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:30.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:30.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 495486634 0 0) 0x7f5cac0ca490 con 0x7f5cac02b4c0 2024-03-29T11:36:30.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 >> v1:172.21.15.22:6790/0 conn(0x7f5cac00b8a0 legacy=0x7f5cac0c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:30.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.854+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 >> v1:172.21.15.204:6789/0 conn(0x7f5cac02a5a0 legacy=0x7f5cac0b8e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:30.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.856+0000 7f5cb0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:30.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.856+0000 7f5cb0ff9640 1 monclient: found mon.a 2024-03-29T11:36:30.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.856+0000 7f5cb0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:30.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.856+0000 7f5cb0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:30.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.856+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cac0b9570 con 0x7f5cac02b4c0 2024-03-29T11:36:30.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.856+0000 7f5cb0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:30.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.856+0000 7f5cb0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:30.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.856+0000 7f5cb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 5 monclient: authenticate success, global_id 4305 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 10 monclient: _renew_subs 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cac0b97c0 con 0x7f5cac02b4c0 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 10 monclient: _renew_subs 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cac0c9b30 con 0x7f5cac02b4c0 2024-03-29T11:36:30.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.859+0000 7f5cb9477640 1 librados: init done 2024-03-29T11:36:30.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.860+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ca8001c80 con 0x7f5cac02b4c0 2024-03-29T11:36:30.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.860+0000 7f5cb0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:30.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.860+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5ca8003a60 con 0x7f5cac02b4c0 2024-03-29T11:36:30.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.860+0000 7f5cb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:30.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.860+0000 7f5cb0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:30.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.860+0000 7f5cb0ff9640 10 monclient: dump: 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:30.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.860+0000 7f5cb27fc640 4 set_mon_vals no callback set 2024-03-29T11:36:30.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.863+0000 7f5cb9477640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:30.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.863+0000 7f5cb9477640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:30.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.863+0000 7f5cb9477640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:30.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.863+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c94005080 con 0x7f5cac02b4c0 2024-03-29T11:36:30.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.863+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f5ca8003bc0 con 0x7f5cac02b4c0 2024-03-29T11:36:30.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.866+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f5ca8084b80 con 0x7f5cac02b4c0 2024-03-29T11:36:30.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.866+0000 7f5cb0ff9640 10 client.4305.objecter ms_dispatch 0x7f5cb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:30.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.866+0000 7f5cb0ff9640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:30.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.866+0000 7f5cb0ff9640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:30.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.866+0000 7f5cb0ff9640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-29T11:36:30.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.866+0000 7f5cb3fff640 10 monclient: get_auth_request con 0x7f5c9806dc90 auth_method 0 2024-03-29T11:36:30.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.867+0000 7f5cb0ff9640 10 client.4305.objecter ms_handle_connect 0x7f5c9806dc90 2024-03-29T11:36:30.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.873+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ca8050460 con 0x7f5cac02b4c0 2024-03-29T11:36:30.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.873+0000 7f5cb0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:30.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:30.873+0000 7f5cb0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:31.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.198+0000 7f5cb9477640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.198+0000 7f5cb9477640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.198+0000 7f5cb9477640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:36:31.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.198+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c94005330 con 0x7f5cac02b4c0 2024-03-29T11:36:31.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.201+0000 7f5cb0ff9640 1 -- 172.21.15.22:0/2115753699 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f5ca8052390 con 0x7f5cac02b4c0 2024-03-29T11:36:31.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.201+0000 7f5cb0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.201+0000 7f5cb0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:31.200 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:36:31.201 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:31.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.203+0000 7f5cb9477640 10 librados: watch_flush enter 2024-03-29T11:36:31.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 10 librados: watch_flush exit 2024-03-29T11:36:31.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-29T11:36:31.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 10 client.4305.objecter successfully canceled tick 2024-03-29T11:36:31.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 >> v1:172.21.15.204:6816/3848099522 conn(0x7f5c9806dc90 legacy=0x7f5c98070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 10 monclient: shutdown 2024-03-29T11:36:31.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:31.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 >> v1:172.21.15.22:6789/0 conn(0x7f5cac02b4c0 legacy=0x7f5cac17a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 shutdown_connections 2024-03-29T11:36:31.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.204+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 >> 172.21.15.22:0/2115753699 conn(0x7f5cac0058f0 msgr2=0x7f5cac0c0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:31.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.207+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 shutdown_connections 2024-03-29T11:36:31.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.207+0000 7f5cb9477640 1 -- 172.21.15.22:0/2115753699 wait complete. 2024-03-29T11:36:31.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:31.207+0000 7f5cb9477640 1 librados: shutdown 2024-03-29T11:36:31.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:38.034 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:38.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.190+0000 7fedc57e8640 10 monclient: get_monmap_and_config 2024-03-29T11:36:38.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.190+0000 7fedc57e8640 10 monclient: build_initial_monmap 2024-03-29T11:36:38.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.190+0000 7fedc57e8640 10 monclient: monmap: 2024-03-29T11:36:38.190 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:38.190 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:38.190 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:36:38.191576+0000 2024-03-29T11:36:38.190 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:36:38.191576+0000 2024-03-29T11:36:38.190 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:38.190 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:38.191 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:38.191 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:36:38.191 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:36:38.191 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:38.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.191+0000 7fedc57e8640 1 Processor -- start 2024-03-29T11:36:38.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.191+0000 7fedc57e8640 1 -- start start 2024-03-29T11:36:38.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.191+0000 7fedc57e8640 10 monclient: init 2024-03-29T11:36:38.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:38.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:38.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 10 monclient(hunting): picked mon.noname-c con 0x7fedc01457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:38.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 10 monclient(hunting): picked mon.noname-a con 0x7fedc013f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:38.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 10 monclient(hunting): picked mon.noname-b con 0x7fedc00a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:38.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00aaef0 con 0x7fedc013f650 2024-03-29T11:36:38.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc0145bd0 con 0x7fedc01457f0 2024-03-29T11:36:38.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc0145eb0 con 0x7fedc00a9d70 2024-03-29T11:36:38.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:38.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.192+0000 7fedc57e8640 10 monclient(hunting): authenticate will time out at 1048.469871s 2024-03-29T11:36:38.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.195+0000 7fedbe7fc640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fedc013f650 0x7fedc00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:54727/0 (socket says 172.21.15.22:54727) 2024-03-29T11:36:38.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.195+0000 7fedbe7fc640 1 -- 172.21.15.22:0/2146461293 learned_addr learned my addr 172.21.15.22:0/2146461293 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:38.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473601698 0 0) 0x7fedc00aaef0 con 0x7fedc013f650 2024-03-29T11:36:38.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:36:38.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda8003520 con 0x7fedc013f650 2024-03-29T11:36:38.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600010494 0 0) 0x7fedc0145eb0 con 0x7fedc00a9d70 2024-03-29T11:36:38.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:36:38.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.196+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc00aaef0 con 0x7fedc00a9d70 2024-03-29T11:36:38.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2853499398 0 0) 0x7feda8003520 con 0x7fedc013f650 2024-03-29T11:36:38.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 >> v1:172.21.15.22:6790/0 conn(0x7fedc00a9d70 legacy=0x7fedc014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 >> v1:172.21.15.204:6789/0 conn(0x7fedc01457f0 legacy=0x7fedc013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:38.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 1 monclient: found mon.noname-a 2024-03-29T11:36:38.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:38.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:36:38.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc0146340 con 0x7fedc013f650 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedc57e8640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedc57e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fedb8001a10 con 0x7fedc013f650 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.198+0000 7fedbdffb640 10 monclient: dump: 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.200 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedc57e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedb8001c30 con 0x7fedc013f650 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedc57e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedc57e8640 4 set_mon_vals no callback set 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedbdffb640 1 -- 172.21.15.22:0/2146461293 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fedb8001fc0 con 0x7fedc013f650 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedbdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.199+0000 7fedbdffb640 10 monclient: dump: 2024-03-29T11:36:38.201 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.200+0000 7fedc57e8640 10 monclient: shutdown 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.200+0000 7fedc57e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:38.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.200+0000 7fedc57e8640 1 -- 172.21.15.22:0/2146461293 >> v1:172.21.15.22:6789/0 conn(0x7fedc013f650 legacy=0x7fedc00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.200+0000 7fedc57e8640 1 -- 172.21.15.22:0/2146461293 shutdown_connections 2024-03-29T11:36:38.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.200+0000 7fedc57e8640 1 -- 172.21.15.22:0/2146461293 >> 172.21.15.22:0/2146461293 conn(0x7fedc0066a00 msgr2=0x7fedc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.204+0000 7fedc57e8640 1 -- 172.21.15.22:0/2146461293 shutdown_connections 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.204+0000 7fedc57e8640 1 -- 172.21.15.22:0/2146461293 wait complete. 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.205+0000 7fedc57e8640 10 monclient: build_initial_monmap 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.205+0000 7fedc57e8640 10 monclient: monmap: 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:36:38.205 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:38.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.206+0000 7fedc57e8640 1 librados: starting msgr at 2024-03-29T11:36:38.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.206+0000 7fedc57e8640 1 librados: starting objecter 2024-03-29T11:36:38.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.206+0000 7fedc57e8640 1 Processor -- start 2024-03-29T11:36:38.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.206+0000 7fedc57e8640 1 -- start start 2024-03-29T11:36:38.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.206+0000 7fedc57e8640 1 librados: setting wanted keys 2024-03-29T11:36:38.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.206+0000 7fedc57e8640 1 librados: calling monclient init 2024-03-29T11:36:38.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.206+0000 7fedc57e8640 10 monclient: init 2024-03-29T11:36:38.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:38.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:38.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 10 monclient(hunting): picked mon.noname-c con 0x7fedc00a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:38.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 10 monclient(hunting): picked mon.noname-a con 0x7fedc013f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:38.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 10 monclient(hunting): picked mon.noname-b con 0x7fedc0071d80 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:38.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc006fba0 con 0x7fedc013f650 2024-03-29T11:36:38.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc006fe10 con 0x7fedc0071d80 2024-03-29T11:36:38.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc013bbf0 con 0x7fedc00a9d70 2024-03-29T11:36:38.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:38.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.207+0000 7fedc57e8640 10 monclient(hunting): authenticate will time out at 1048.484666s 2024-03-29T11:36:38.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.210+0000 7fedbeffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fedc00a9d70 0x7fedc006e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47027/0 (socket says 172.21.15.22:47027) 2024-03-29T11:36:38.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.210+0000 7fedbeffd640 1 -- 172.21.15.22:0/639860175 learned_addr learned my addr 172.21.15.22:0/639860175 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:38.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.213+0000 7feda77fe640 10 client.?.objecter ms_handle_connect 0x7fedc00a9d70 2024-03-29T11:36:38.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.213+0000 7feda77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 client.?.objecter ms_handle_connect 0x7fedc0071d80 2024-03-29T11:36:38.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181031916 0 0) 0x7fedc013bbf0 con 0x7fedc00a9d70 2024-03-29T11:36:38.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:36:38.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac004eb0 con 0x7fedc00a9d70 2024-03-29T11:36:38.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 client.?.objecter ms_handle_connect 0x7fedc013f650 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257006781 0 0) 0x7fedc006fe10 con 0x7fedc0071d80 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc013bbf0 con 0x7fedc0071d80 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811181029 0 0) 0x7fedc006fba0 con 0x7fedc013f650 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc006fe10 con 0x7fedc013f650 2024-03-29T11:36:38.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455329859 0 0) 0x7fedc013bbf0 con 0x7fedc0071d80 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedc006fba0 con 0x7fedc0071d80 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1581235590 0 0) 0x7fedac004eb0 con 0x7fedc00a9d70 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedc013bbf0 con 0x7fedc00a9d70 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fedb00018f0 con 0x7fedc0071d80 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): dump: 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:38.216 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fedb4001950 con 0x7fedc00a9d70 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 10 monclient(hunting): dump: 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:38.217 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 354519477 0 0) 0x7fedc006fe10 con 0x7fedc013f650 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.214+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedac004eb0 con 0x7fedc013f650 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fedb8001490 con 0x7fedc013f650 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 10 monclient(hunting): dump: 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.218 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 323124594 0 0) 0x7fedc006fba0 con 0x7fedc0071d80 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 >> v1:172.21.15.22:6790/0 conn(0x7fedc00a9d70 legacy=0x7fedc006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 >> v1:172.21.15.22:6789/0 conn(0x7fedc013f650 legacy=0x7fedc0071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 1 monclient: found mon.b 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc006e810 con 0x7fedc0071d80 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:38.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7feda77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 10 monclient: _renew_subs 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc01e48a0 con 0x7fedc0071d80 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 10 monclient: _renew_subs 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc01e4d30 con 0x7fedc0071d80 2024-03-29T11:36:38.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.215+0000 7fedc57e8640 1 librados: init done 2024-03-29T11:36:38.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.216+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedb00035f0 con 0x7fedc0071d80 2024-03-29T11:36:38.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.216+0000 7feda77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:38.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.216+0000 7fedbcff9640 4 set_mon_vals no callback set 2024-03-29T11:36:38.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.216+0000 7fedc57e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.217+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fedb0003900 con 0x7fedc0071d80 2024-03-29T11:36:38.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.217+0000 7feda77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.217+0000 7feda77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:38.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.217+0000 7feda77fe640 10 monclient: dump: 2024-03-29T11:36:38.222 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:38.222 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:38.222 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.222 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.217+0000 7fedc57e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.217+0000 7fedc57e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.217+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda0005080 con 0x7fedc0071d80 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.221+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fedb00198d0 con 0x7fedc0071d80 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.221+0000 7fedbeffd640 10 monclient: get_auth_request con 0x7fedac06e0d0 auth_method 0 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.222+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fedb00836c0 con 0x7fedc0071d80 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.222+0000 7feda77fe640 10 client.4309.objecter ms_dispatch 0x7fedc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:38.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.222+0000 7feda77fe640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:38.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.222+0000 7feda77fe640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:38.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.222+0000 7feda77fe640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-29T11:36:38.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.222+0000 7feda77fe640 10 client.4309.objecter ms_handle_connect 0x7fedac06e0d0 2024-03-29T11:36:38.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.225+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fedb004efa0 con 0x7fedc0071d80 2024-03-29T11:36:38.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.226+0000 7feda77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.226+0000 7feda77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.545+0000 7fedc57e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.545+0000 7fedc57e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.545+0000 7fedc57e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.545+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feda0005330 con 0x7fedc0071d80 2024-03-29T11:36:38.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.548+0000 7feda77fe640 1 -- 172.21.15.22:0/639860175 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fedb0050ed0 con 0x7fedc0071d80 2024-03-29T11:36:38.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.548+0000 7feda77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:38.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.548+0000 7feda77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:38.547 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:36:38.548 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:38.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.551+0000 7fedc57e8640 10 librados: watch_flush enter 2024-03-29T11:36:38.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 10 librados: watch_flush exit 2024-03-29T11:36:38.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-29T11:36:38.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 10 client.4309.objecter successfully canceled tick 2024-03-29T11:36:38.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 >> v1:172.21.15.204:6816/3848099522 conn(0x7fedac06e0d0 legacy=0x7fedac070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 10 monclient: shutdown 2024-03-29T11:36:38.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:38.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 >> v1:172.21.15.204:6789/0 conn(0x7fedc0071d80 legacy=0x7fedc01e4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 shutdown_connections 2024-03-29T11:36:38.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.552+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 >> 172.21.15.22:0/639860175 conn(0x7fedc0066a00 msgr2=0x7fedc00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:38.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.553+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 shutdown_connections 2024-03-29T11:36:38.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.553+0000 7fedc57e8640 1 -- 172.21.15.22:0/639860175 wait complete. 2024-03-29T11:36:38.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:38.553+0000 7fedc57e8640 1 librados: shutdown 2024-03-29T11:36:38.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:45.381 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.519+0000 7f08d24f6640 10 monclient: get_monmap_and_config 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.519+0000 7f08d24f6640 10 monclient: build_initial_monmap 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.519+0000 7f08d24f6640 10 monclient: monmap: 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:36:45.520458+0000 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:36:45.520458+0000 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:36:45.519 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.520+0000 7f08d24f6640 1 Processor -- start 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.520+0000 7f08d24f6640 1 -- start start 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.520+0000 7f08d24f6640 10 monclient: init 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.520+0000 7f08d24f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.520+0000 7f08d24f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.520+0000 7f08d24f6640 10 monclient(hunting): picked mon.noname-a con 0x7f08cc1457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.520+0000 7f08d24f6640 10 monclient(hunting): picked mon.noname-c con 0x7f08cc13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08d24f6640 10 monclient(hunting): picked mon.noname-b con 0x7f08cc0a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08d24f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc0aaef0 con 0x7f08cc1457f0 2024-03-29T11:36:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08d24f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc145bd0 con 0x7f08cc13f650 2024-03-29T11:36:45.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08d24f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc145eb0 con 0x7f08cc0a9d70 2024-03-29T11:36:45.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08d24f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:45.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08d24f6640 10 monclient(hunting): authenticate will time out at 1055.798110s 2024-03-29T11:36:45.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08cbfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f08cc1457f0 0x7f08cc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58756/0 (socket says 172.21.15.22:58756) 2024-03-29T11:36:45.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08cbfff640 1 -- 172.21.15.22:0/930886476 learned_addr learned my addr 172.21.15.22:0/930886476 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:45.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052971429 0 0) 0x7f08cc0aaef0 con 0x7f08cc1457f0 2024-03-29T11:36:45.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:45.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:45.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:45.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08caffd640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:36:45.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.521+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b4003520 con 0x7f08cc1457f0 2024-03-29T11:36:45.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.523+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666498854 0 0) 0x7f08b4003520 con 0x7f08cc1457f0 2024-03-29T11:36:45.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.523+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 >> v1:172.21.15.22:6790/0 conn(0x7f08cc0a9d70 legacy=0x7f08cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:45.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.523+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 >> v1:172.21.15.204:6789/0 conn(0x7f08cc13f650 legacy=0x7f08cc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:45.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 1 monclient: found mon.noname-a 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc146340 con 0x7f08cc1457f0 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08bc001670 con 0x7f08cc1457f0 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: dump: 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:45.525 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:45.526 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:45.526 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:45.526 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:45.526 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:45.526 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:45.526 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:45.526 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:45.527 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:45.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08d24f6640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:36:45.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08d24f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08bc001840 con 0x7f08cc1457f0 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 1 -- 172.21.15.22:0/930886476 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08bc001c90 con 0x7f08cc1457f0 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08caffd640 10 monclient: dump: 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:45.531 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08d24f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.524+0000 7f08d24f6640 4 set_mon_vals no callback set 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.525+0000 7f08d24f6640 10 monclient: shutdown 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.525+0000 7f08d24f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:45.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.525+0000 7f08d24f6640 1 -- 172.21.15.22:0/930886476 >> v1:172.21.15.22:6789/0 conn(0x7f08cc1457f0 legacy=0x7f08cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.525+0000 7f08d24f6640 1 -- 172.21.15.22:0/930886476 shutdown_connections 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.525+0000 7f08d24f6640 1 -- 172.21.15.22:0/930886476 >> 172.21.15.22:0/930886476 conn(0x7f08cc066a00 msgr2=0x7f08cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.526+0000 7f08d24f6640 1 -- 172.21.15.22:0/930886476 shutdown_connections 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.526+0000 7f08d24f6640 1 -- 172.21.15.22:0/930886476 wait complete. 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.527+0000 7f08d24f6640 10 monclient: build_initial_monmap 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.527+0000 7f08d24f6640 10 monclient: monmap: 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:45.533 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.527+0000 7f08d24f6640 1 librados: starting msgr at 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.527+0000 7f08d24f6640 1 librados: starting objecter 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.528+0000 7f08d24f6640 1 Processor -- start 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.528+0000 7f08d24f6640 1 -- start start 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.528+0000 7f08d24f6640 1 librados: setting wanted keys 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.528+0000 7f08d24f6640 1 librados: calling monclient init 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.528+0000 7f08d24f6640 10 monclient: init 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:45.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 10 monclient(hunting): picked mon.noname-b con 0x7f08cc0a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 10 monclient(hunting): picked mon.noname-a con 0x7f08cc13f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08cb7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f08cc13f650 0x7f08cc1422d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58952/0 (socket says 172.21.15.22:58952) 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d0a93640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f08cc1457f0 0x7f08cc070720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58912/0 (socket says 172.21.15.22:58912) 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 10 monclient(hunting): picked mon.noname-c con 0x7f08cc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc071b60 con 0x7f08cc13f650 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc071dd0 con 0x7f08cc0a9d70 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc070e30 con 0x7f08cc1457f0 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08d24f6640 10 monclient(hunting): authenticate will time out at 1055.806602s 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08cb7fe640 1 -- 172.21.15.22:0/3941055964 learned_addr learned my addr 172.21.15.22:0/3941055964 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08c8ff9640 10 client.?.objecter ms_handle_connect 0x7f08cc1457f0 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:45.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08c8ff9640 10 client.?.objecter ms_handle_connect 0x7f08cc13f650 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507997666 0 0) 0x7f08cc070e30 con 0x7f08cc1457f0 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.529+0000 7f08c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b8004eb0 con 0x7f08cc1457f0 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762905867 0 0) 0x7f08b8004eb0 con 0x7f08cc1457f0 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08cc070e30 con 0x7f08cc1457f0 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08c4001ba0 con 0x7f08cc1457f0 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:45.536 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604526812 0 0) 0x7f08cc071b60 con 0x7f08cc13f650 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:36:45.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b8004eb0 con 0x7f08cc13f650 2024-03-29T11:36:45.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1842088365 0 0) 0x7f08cc070e30 con 0x7f08cc1457f0 2024-03-29T11:36:45.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.530+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 >> v1:172.21.15.204:6789/0 conn(0x7f08cc0a9d70 legacy=0x7f08cc06f9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:45.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 >> v1:172.21.15.22:6789/0 conn(0x7f08cc13f650 legacy=0x7f08cc1422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:45.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:45.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 1 monclient: found mon.c 2024-03-29T11:36:45.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:45.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc070fc0 con 0x7f08cc1457f0 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08c4001d00 con 0x7f08cc1457f0 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08c4003b50 con 0x7f08cc1457f0 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.532+0000 7f08c8ff9640 10 monclient: dump: 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:45.542 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 10 monclient: _renew_subs 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc06de90 con 0x7f08cc1457f0 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:45.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 10 monclient: _renew_subs 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc06e6f0 con 0x7f08cc1457f0 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08d24f6640 1 librados: init done 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.533+0000 7f08ca7fc640 4 set_mon_vals no callback set 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.535+0000 7f08d24f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.535+0000 7f08d24f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.535+0000 7f08d24f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.535+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc136c10 con 0x7f08cc1457f0 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.535+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f08c4003cb0 con 0x7f08cc1457f0 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.536+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f08c4084cd0 con 0x7f08cc1457f0 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.536+0000 7f08c8ff9640 10 client.4301.objecter ms_dispatch 0x7f08cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.536+0000 7f08c8ff9640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.536+0000 7f08c8ff9640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.536+0000 7f08c8ff9640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:36:45.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.536+0000 7f08cbfff640 10 monclient: get_auth_request con 0x7f08b806daf0 auth_method 0 2024-03-29T11:36:45.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.537+0000 7f08c8ff9640 10 client.4301.objecter ms_handle_connect 0x7f08b806daf0 2024-03-29T11:36:45.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.543+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08cc136c10 con 0x7f08cc1457f0 2024-03-29T11:36:45.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.543+0000 7f08c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:45.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.544+0000 7f08c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:45.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.868+0000 7f08d24f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:45.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.868+0000 7f08d24f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:45.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.868+0000 7f08d24f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:36:45.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.868+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08cc136c10 con 0x7f08cc1457f0 2024-03-29T11:36:45.870 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:36:45.870 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:45.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.869+0000 7f08c8ff9640 1 -- 172.21.15.22:0/3941055964 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f08cc136c10 con 0x7f08cc1457f0 2024-03-29T11:36:45.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.869+0000 7f08c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:45.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.869+0000 7f08c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:45.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 10 librados: watch_flush enter 2024-03-29T11:36:45.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 10 librados: watch_flush exit 2024-03-29T11:36:45.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:36:45.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:36:45.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 >> v1:172.21.15.204:6816/3848099522 conn(0x7f08b806daf0 legacy=0x7f08b806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:45.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 10 monclient: shutdown 2024-03-29T11:36:45.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:45.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 >> v1:172.21.15.22:6790/0 conn(0x7f08cc1457f0 legacy=0x7f08cc070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:45.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 shutdown_connections 2024-03-29T11:36:45.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 >> 172.21.15.22:0/3941055964 conn(0x7f08cc066a00 msgr2=0x7f08cc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:45.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 shutdown_connections 2024-03-29T11:36:45.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.872+0000 7f08d24f6640 1 -- 172.21.15.22:0/3941055964 wait complete. 2024-03-29T11:36:45.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:45.873+0000 7f08d24f6640 1 librados: shutdown 2024-03-29T11:36:45.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:52.694 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:52.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.838+0000 7fe114310640 10 monclient: get_monmap_and_config 2024-03-29T11:36:52.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.838+0000 7fe114310640 10 monclient: build_initial_monmap 2024-03-29T11:36:52.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.838+0000 7fe114310640 10 monclient: monmap: 2024-03-29T11:36:52.838 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:52.838 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:52.838 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:36:52.839650+0000 2024-03-29T11:36:52.838 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:36:52.839650+0000 2024-03-29T11:36:52.838 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:52.839 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:52.839 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:52.839 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:36:52.839 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:36:52.839 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.839+0000 7fe114310640 1 Processor -- start 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.839+0000 7fe114310640 1 -- start start 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.839+0000 7fe114310640 10 monclient: init 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c1457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c13f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 10 monclient(hunting): picked mon.noname-a con 0x7fe10c0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0aaef0 con 0x7fe10c0a9d70 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c145bd0 con 0x7fe10c1457f0 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c145eb0 con 0x7fe10c13f650 2024-03-29T11:36:52.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:52.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe114310640 10 monclient(hunting): authenticate will time out at 1063.117465s 2024-03-29T11:36:52.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe1118ab640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe10c13f650 0x7fe10c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35308/0 (socket says 172.21.15.22:35308) 2024-03-29T11:36:52.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.840+0000 7fe1118ab640 1 -- 172.21.15.22:0/3949616390 learned_addr learned my addr 172.21.15.22:0/3949616390 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622718137 0 0) 0x7fe10c145bd0 con 0x7fe10c1457f0 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f8003520 con 0x7fe10c1457f0 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778598423 0 0) 0x7fe10c145eb0 con 0x7fe10c13f650 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10c145bd0 con 0x7fe10c13f650 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3418251738 0 0) 0x7fe0f8003520 con 0x7fe10c1457f0 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 >> v1:172.21.15.22:6790/0 conn(0x7fe10c13f650 legacy=0x7fe10c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 >> v1:172.21.15.22:6789/0 conn(0x7fe10c0a9d70 legacy=0x7fe10c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 monclient: found mon.noname-c 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c146340 con 0x7fe10c1457f0 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient: _finish_auth 0 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:52.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe100001690 con 0x7fe10c1457f0 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.841+0000 7fe1110aa640 10 monclient: dump: 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe114310640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:36:52.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe114310640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:52.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe100001870 con 0x7fe10c1457f0 2024-03-29T11:36:52.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe1110aa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:52.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe114310640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe114310640 4 set_mon_vals no callback set 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe1110aa640 1 -- 172.21.15.22:0/3949616390 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe100001c80 con 0x7fe10c1457f0 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe1110aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe1110aa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.842+0000 7fe1110aa640 10 monclient: dump: 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:52.853 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.843+0000 7fe114310640 10 monclient: shutdown 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.843+0000 7fe114310640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.843+0000 7fe114310640 1 -- 172.21.15.22:0/3949616390 >> v1:172.21.15.204:6789/0 conn(0x7fe10c1457f0 legacy=0x7fe10c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.843+0000 7fe114310640 1 -- 172.21.15.22:0/3949616390 shutdown_connections 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.843+0000 7fe114310640 1 -- 172.21.15.22:0/3949616390 >> 172.21.15.22:0/3949616390 conn(0x7fe10c066a00 msgr2=0x7fe10c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.843+0000 7fe114310640 1 -- 172.21.15.22:0/3949616390 shutdown_connections 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.843+0000 7fe114310640 1 -- 172.21.15.22:0/3949616390 wait complete. 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.844+0000 7fe114310640 10 monclient: build_initial_monmap 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.844+0000 7fe114310640 10 monclient: monmap: 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:52.854 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.845+0000 7fe114310640 1 librados: starting msgr at 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.845+0000 7fe114310640 1 librados: starting objecter 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.845+0000 7fe114310640 1 Processor -- start 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.845+0000 7fe114310640 1 -- start start 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.845+0000 7fe114310640 1 librados: setting wanted keys 2024-03-29T11:36:52.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.845+0000 7fe114310640 1 librados: calling monclient init 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.845+0000 7fe114310640 10 monclient: init 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c0a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 10 monclient(hunting): picked mon.noname-a con 0x7fe10c13f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c1457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c137890 con 0x7fe10c13f650 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06f4b0 con 0x7fe10c1457f0 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06f790 con 0x7fe10c0a9d70 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.846+0000 7fe114310640 10 monclient(hunting): authenticate will time out at 1063.123620s 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.847+0000 7fe1128ad640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fe10c1457f0 0x7fe10c137180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:49046/0 (socket says 172.21.15.22:49046) 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.847+0000 7fe1128ad640 1 -- 172.21.15.22:0/3914967617 learned_addr learned my addr 172.21.15.22:0/3914967617 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:36:52.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.847+0000 7fe0feffd640 10 client.?.objecter ms_handle_connect 0x7fe10c1457f0 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.847+0000 7fe0feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.847+0000 7fe0feffd640 10 client.?.objecter ms_handle_connect 0x7fe10c13f650 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.847+0000 7fe0feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120670224 0 0) 0x7fe10c137890 con 0x7fe10c13f650 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0004eb0 con 0x7fe10c13f650 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789699204 0 0) 0x7fe10c06f4b0 con 0x7fe10c1457f0 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10c137890 con 0x7fe10c1457f0 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 client.?.objecter ms_handle_connect 0x7fe10c0a9d70 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304185087 0 0) 0x7fe10c06f790 con 0x7fe10c0a9d70 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.848+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10c06f4b0 con 0x7fe10c0a9d70 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1917546844 0 0) 0x7fe0f0004eb0 con 0x7fe10c13f650 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe10c06f790 con 0x7fe10c13f650 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe108001930 con 0x7fe10c13f650 2024-03-29T11:36:52.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): dump: 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:52.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4168080720 0 0) 0x7fe10c137890 con 0x7fe10c1457f0 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f0004eb0 con 0x7fe10c1457f0 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe104001cb0 con 0x7fe10c1457f0 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): dump: 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:52.860 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4284901194 0 0) 0x7fe10c06f4b0 con 0x7fe10c0a9d70 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe10c137890 con 0x7fe10c0a9d70 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe100001c80 con 0x7fe10c0a9d70 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 10 monclient(hunting): dump: 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.861 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1867127230 0 0) 0x7fe0f0004eb0 con 0x7fe10c1457f0 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 >> v1:172.21.15.22:6790/0 conn(0x7fe10c0a9d70 legacy=0x7fe10c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.849+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 >> v1:172.21.15.22:6789/0 conn(0x7fe10c13f650 legacy=0x7fe10c136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe0feffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe0feffd640 1 monclient: found mon.b 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe0feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe0feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c06e2e0 con 0x7fe10c1457f0 2024-03-29T11:36:52.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe0feffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe0feffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe0feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 10 monclient: _renew_subs 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c142d50 con 0x7fe10c1457f0 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 10 monclient: _renew_subs 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c06de90 con 0x7fe10c1457f0 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.850+0000 7fe114310640 1 librados: init done 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.851+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe104001e10 con 0x7fe10c1457f0 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.851+0000 7fe0feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:52.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.851+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe104003bf0 con 0x7fe10c1457f0 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.851+0000 7fe0feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.851+0000 7fe0feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.851+0000 7fe1108a9640 4 set_mon_vals no callback set 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.851+0000 7fe0feffd640 10 monclient: dump: 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:36:52.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.852+0000 7fe114310640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:52.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.852+0000 7fe114310640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:52.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.852+0000 7fe114310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:52.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.852+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f4005080 con 0x7fe10c1457f0 2024-03-29T11:36:52.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.854+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fe104003e30 con 0x7fe10c1457f0 2024-03-29T11:36:52.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.855+0000 7fe1120ac640 10 monclient: get_auth_request con 0x7fe0f006e0a0 auth_method 0 2024-03-29T11:36:52.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.855+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fe104084de0 con 0x7fe10c1457f0 2024-03-29T11:36:52.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.855+0000 7fe0feffd640 10 client.4315.objecter ms_dispatch 0x7fe10c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:52.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.855+0000 7fe0feffd640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:52.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.855+0000 7fe0feffd640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:52.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.855+0000 7fe0feffd640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-29T11:36:52.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.855+0000 7fe0feffd640 10 client.4315.objecter ms_handle_connect 0x7fe0f006e0a0 2024-03-29T11:36:52.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.862+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1040506c0 con 0x7fe10c1457f0 2024-03-29T11:36:52.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.862+0000 7fe0feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:52.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:52.862+0000 7fe0feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:53.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.270+0000 7fe114310640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.270+0000 7fe114310640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.270+0000 7fe114310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:36:53.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.270+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0f4005330 con 0x7fe10c1457f0 2024-03-29T11:36:53.271 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:36:53.271 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:53.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.271+0000 7fe0feffd640 1 -- 172.21.15.22:0/3914967617 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fe1040525f0 con 0x7fe10c1457f0 2024-03-29T11:36:53.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.271+0000 7fe0feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.271+0000 7fe0feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:53.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 10 librados: watch_flush enter 2024-03-29T11:36:53.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 10 librados: watch_flush exit 2024-03-29T11:36:53.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-29T11:36:53.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 10 client.4315.objecter successfully canceled tick 2024-03-29T11:36:53.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 >> v1:172.21.15.204:6816/3848099522 conn(0x7fe0f006e0a0 legacy=0x7fe0f0070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 10 monclient: shutdown 2024-03-29T11:36:53.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:53.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 >> v1:172.21.15.204:6789/0 conn(0x7fe10c1457f0 legacy=0x7fe10c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 shutdown_connections 2024-03-29T11:36:53.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 >> 172.21.15.22:0/3914967617 conn(0x7fe10c066a00 msgr2=0x7fe10c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:53.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.275+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 shutdown_connections 2024-03-29T11:36:53.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.276+0000 7fe114310640 1 -- 172.21.15.22:0/3914967617 wait complete. 2024-03-29T11:36:53.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:36:53.276+0000 7fe114310640 1 librados: shutdown 2024-03-29T11:36:53.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:00.097 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.243+0000 7f5e8e4ff640 10 monclient: get_monmap_and_config 2024-03-29T11:37:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.243+0000 7f5e8e4ff640 10 monclient: build_initial_monmap 2024-03-29T11:37:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.243+0000 7f5e8e4ff640 10 monclient: monmap: 2024-03-29T11:37:00.245 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:00.244223+0000 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:00.244223+0000 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:00.246 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.247+0000 7f5e8e4ff640 1 Processor -- start 2024-03-29T11:37:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.247+0000 7f5e8e4ff640 1 -- start start 2024-03-29T11:37:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.247+0000 7f5e8e4ff640 10 monclient: init 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 10 monclient(hunting): picked mon.noname-b con 0x7f5e8806d9d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 10 monclient(hunting): picked mon.noname-c con 0x7f5e8813ef30 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 10 monclient(hunting): picked mon.noname-a con 0x7f5e880b0860 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880aed10 con 0x7f5e880b0860 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880ab1b0 con 0x7f5e8813ef30 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880ab490 con 0x7f5e8806d9d0 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e8e4ff640 10 monclient(hunting): authenticate will time out at 1070.524165s 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e87fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5e8806d9d0 0x7f5e8813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37376/0 (socket says 172.21.15.22:37376) 2024-03-29T11:37:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.248+0000 7f5e87fff640 1 -- 172.21.15.22:0/209530954 learned_addr learned my addr 172.21.15.22:0/209530954 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:00.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493750810 0 0) 0x7f5e880aed10 con 0x7f5e880b0860 2024-03-29T11:37:00.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e70003520 con 0x7f5e880b0860 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157730832 0 0) 0x7f5e880ab490 con 0x7f5e8806d9d0 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e880aed10 con 0x7f5e8806d9d0 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455382801 0 0) 0x7f5e70003520 con 0x7f5e880b0860 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 >> v1:172.21.15.22:6790/0 conn(0x7f5e8806d9d0 legacy=0x7f5e8813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.251+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 >> v1:172.21.15.204:6789/0 conn(0x7f5e8813ef30 legacy=0x7f5e880af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:00.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 1 monclient: found mon.noname-a 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880ab770 con 0x7f5e880b0860 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5e7c001690 con 0x7f5e880b0860 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: dump: 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.253 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e7c0017f0 con 0x7f5e880b0860 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 1 -- 172.21.15.22:0/209530954 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5e7c001bc0 con 0x7f5e880b0860 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.252+0000 7f5e86ffd640 10 monclient: dump: 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.254 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.255 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.255 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:00.255 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:00.255 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:00.255 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:00.255 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.253+0000 7f5e8e4ff640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.253+0000 7f5e8e4ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.253+0000 7f5e8e4ff640 4 set_mon_vals no callback set 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.255+0000 7f5e8e4ff640 10 monclient: shutdown 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.255+0000 7f5e8e4ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.255+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/209530954 >> v1:172.21.15.22:6789/0 conn(0x7f5e880b0860 legacy=0x7f5e88142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.258+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/209530954 shutdown_connections 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.258+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/209530954 >> 172.21.15.22:0/209530954 conn(0x7f5e8806b660 msgr2=0x7f5e8806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:00.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.259+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/209530954 shutdown_connections 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.259+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/209530954 wait complete. 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.260+0000 7f5e8e4ff640 10 monclient: build_initial_monmap 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.260+0000 7f5e8e4ff640 10 monclient: monmap: 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:00.259 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:00.260 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:00.260 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:00.260 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:00.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.260+0000 7f5e8e4ff640 1 librados: starting msgr at 2024-03-29T11:37:00.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.260+0000 7f5e8e4ff640 1 librados: starting objecter 2024-03-29T11:37:00.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.260+0000 7f5e8e4ff640 1 Processor -- start 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.261+0000 7f5e8e4ff640 1 -- start start 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.261+0000 7f5e8e4ff640 1 librados: setting wanted keys 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.261+0000 7f5e8e4ff640 1 librados: calling monclient init 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.261+0000 7f5e8e4ff640 10 monclient: init 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 10 monclient(hunting): picked mon.noname-c con 0x7f5e8806d9d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 10 monclient(hunting): picked mon.noname-b con 0x7f5e880b0860 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 10 monclient(hunting): picked mon.noname-a con 0x7f5e8813ef30 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880b1ab0 con 0x7f5e8813ef30 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880b3bc0 con 0x7f5e880b0860 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880b3ea0 con 0x7f5e8806d9d0 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:00.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e8e4ff640 10 monclient(hunting): authenticate will time out at 1070.538181s 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e87fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5e8806d9d0 0x7f5e880b2850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:38616/0 (socket says 172.21.15.22:38616) 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.262+0000 7f5e87fff640 1 -- 172.21.15.22:0/683571713 learned_addr learned my addr 172.21.15.22:0/683571713 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.263+0000 7f5e84ff9640 10 client.?.objecter ms_handle_connect 0x7f5e8806d9d0 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.263+0000 7f5e84ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.263+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278889194 0 0) 0x7f5e880b3ea0 con 0x7f5e8806d9d0 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.263+0000 7f5e84ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.263+0000 7f5e84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.263+0000 7f5e84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.263+0000 7f5e84ff9640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:37:00.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.263+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e74004eb0 con 0x7f5e8806d9d0 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 client.?.objecter ms_handle_connect 0x7f5e880b0860 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 client.?.objecter ms_handle_connect 0x7f5e8813ef30 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717277696 0 0) 0x7f5e74004eb0 con 0x7f5e8806d9d0 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e880b3ea0 con 0x7f5e8806d9d0 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384837926 0 0) 0x7f5e880b3bc0 con 0x7f5e880b0860 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:37:00.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e74004eb0 con 0x7f5e880b0860 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750814524 0 0) 0x7f5e880b1ab0 con 0x7f5e8813ef30 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e880b3bc0 con 0x7f5e8813ef30 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5e78001990 con 0x7f5e8806d9d0 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:00.273 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 247012848 0 0) 0x7f5e880b3bc0 con 0x7f5e8813ef30 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e880b1ab0 con 0x7f5e8813ef30 2024-03-29T11:37:00.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5e7c001b00 con 0x7f5e8813ef30 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:00.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.267+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 495398816 0 0) 0x7f5e74004eb0 con 0x7f5e880b0860 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e880b3bc0 con 0x7f5e880b0860 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5e80001a10 con 0x7f5e880b0860 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:00.276 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2716160261 0 0) 0x7f5e880b1ab0 con 0x7f5e8813ef30 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 >> v1:172.21.15.22:6790/0 conn(0x7f5e8806d9d0 legacy=0x7f5e880b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 >> v1:172.21.15.204:6789/0 conn(0x7f5e880b0860 legacy=0x7f5e881369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 1 monclient: found mon.a 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880b4180 con 0x7f5e8813ef30 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:37:00.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 10 monclient: _renew_subs 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e880b2f60 con 0x7f5e8813ef30 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 10 monclient: _renew_subs 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8806df90 con 0x7f5e8813ef30 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.268+0000 7f5e8e4ff640 1 librados: init done 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.270+0000 7f5e8e4ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.270+0000 7f5e8e4ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.270+0000 7f5e8e4ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.271+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e7c001c60 con 0x7f5e8813ef30 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.270+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e880b3150 con 0x7f5e8813ef30 2024-03-29T11:37:00.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.271+0000 7f5e84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.271+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5e7c0038a0 con 0x7f5e8813ef30 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.271+0000 7f5e84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.271+0000 7f5e84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.271+0000 7f5e84ff9640 10 monclient: dump: 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:00.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.271+0000 7f5e85ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:00.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.272+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f5e7c019870 con 0x7f5e8813ef30 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.275+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f5e7c083760 con 0x7f5e8813ef30 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.275+0000 7f5e84ff9640 10 client.4326.objecter ms_dispatch 0x7f5e88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.275+0000 7f5e84ff9640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.275+0000 7f5e84ff9640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.275+0000 7f5e84ff9640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.276+0000 7f5e87fff640 10 monclient: get_auth_request con 0x7f5e7406e180 auth_method 0 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.276+0000 7f5e84ff9640 10 client.4326.objecter ms_handle_connect 0x7f5e7406e180 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.280+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e7c04efb0 con 0x7f5e8813ef30 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.280+0000 7f5e84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.280+0000 7f5e84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.612+0000 7f5e8e4ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.613+0000 7f5e8e4ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.613+0000 7f5e8e4ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.613+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e88138110 con 0x7f5e8813ef30 2024-03-29T11:37:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.615+0000 7f5e84ff9640 1 -- 172.21.15.22:0/683571713 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f5e7c050f70 con 0x7f5e8813ef30 2024-03-29T11:37:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.615+0000 7f5e84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.615+0000 7f5e84ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:00.615 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:00.615 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:00.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.619+0000 7f5e8e4ff640 10 librados: watch_flush enter 2024-03-29T11:37:00.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.619+0000 7f5e8e4ff640 10 librados: watch_flush exit 2024-03-29T11:37:00.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.619+0000 7f5e8e4ff640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.619+0000 7f5e8e4ff640 10 client.4326.objecter successfully canceled tick 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 >> v1:172.21.15.204:6816/3848099522 conn(0x7f5e7406e180 legacy=0x7f5e74070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 10 monclient: shutdown 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 >> v1:172.21.15.22:6789/0 conn(0x7f5e8813ef30 legacy=0x7f5e881e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 shutdown_connections 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 >> 172.21.15.22:0/683571713 conn(0x7f5e8806b660 msgr2=0x7f5e8813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 shutdown_connections 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 1 -- 172.21.15.22:0/683571713 wait complete. 2024-03-29T11:37:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:00.620+0000 7f5e8e4ff640 1 librados: shutdown 2024-03-29T11:37:00.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:07.443 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:07.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.594+0000 7fdcabfff640 10 monclient: get_monmap_and_config 2024-03-29T11:37:07.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.594+0000 7fdcabfff640 10 monclient: build_initial_monmap 2024-03-29T11:37:07.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.594+0000 7fdcabfff640 10 monclient: monmap: 2024-03-29T11:37:07.595 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:07.595 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:07.595 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:07.594917+0000 2024-03-29T11:37:07.595 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:07.594917+0000 2024-03-29T11:37:07.595 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:07.595 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:07.595 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:07.596 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:07.596 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:07.596 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:07.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.595+0000 7fdcabfff640 1 Processor -- start 2024-03-29T11:37:07.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.595+0000 7fdcabfff640 1 -- start start 2024-03-29T11:37:07.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.595+0000 7fdcabfff640 10 monclient: init 2024-03-29T11:37:07.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:07.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:07.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 10 monclient(hunting): picked mon.noname-c con 0x7fdca402a490 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:07.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 10 monclient(hunting): picked mon.noname-a con 0x7fdca402b810 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:07.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 10 monclient(hunting): picked mon.noname-b con 0x7fdca4032ce0 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:07.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca402aa40 con 0x7fdca402b810 2024-03-29T11:37:07.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca402ca30 con 0x7fdca402a490 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca402bc70 con 0x7fdca4032ce0 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.596+0000 7fdcabfff640 10 monclient(hunting): authenticate will time out at 1077.872891s 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.599+0000 7fdcab7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fdca4032ce0 0x7fdca40c71e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50508/0 (socket says 172.21.15.22:50508) 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.599+0000 7fdcab7fe640 1 -- 172.21.15.22:0/3054395472 learned_addr learned my addr 172.21.15.22:0/3054395472 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 1 -- 172.21.15.22:0/3054395472 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824554477 0 0) 0x7fdca402bc70 con 0x7fdca4032ce0 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:07.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:07.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:37:07.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 1 -- 172.21.15.22:0/3054395472 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc94003520 con 0x7fdca4032ce0 2024-03-29T11:37:07.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 1 -- 172.21.15.22:0/3054395472 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1602760898 0 0) 0x7fdc94003520 con 0x7fdca4032ce0 2024-03-29T11:37:07.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 1 -- 172.21.15.22:0/3054395472 >> v1:172.21.15.204:6789/0 conn(0x7fdca402a490 legacy=0x7fdca40345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:07.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.600+0000 7fdca9ffb640 1 -- 172.21.15.22:0/3054395472 >> v1:172.21.15.22:6789/0 conn(0x7fdca402b810 legacy=0x7fdca40325d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:07.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:07.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:37:07.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:07.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:37:07.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 1 -- 172.21.15.22:0/3054395472 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca402cca0 con 0x7fdca4032ce0 2024-03-29T11:37:07.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:07.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:07.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:07.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 1 -- 172.21.15.22:0/3054395472 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fdcac04f730 con 0x7fdca4032ce0 2024-03-29T11:37:07.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:07.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.603+0000 7fdca9ffb640 10 monclient: dump: 2024-03-29T11:37:07.605 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.604+0000 7fdcabfff640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:37:07.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.604+0000 7fdcabfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:07.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.604+0000 7fdca9ffb640 1 -- 172.21.15.22:0/3054395472 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcac0008d0 con 0x7fdca4032ce0 2024-03-29T11:37:07.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.604+0000 7fdca9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:07.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.604+0000 7fdcabfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:07.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.604+0000 7fdcabfff640 4 set_mon_vals no callback set 2024-03-29T11:37:07.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.605+0000 7fdcabfff640 10 monclient: shutdown 2024-03-29T11:37:07.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.605+0000 7fdcabfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:07.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.605+0000 7fdcabfff640 1 -- 172.21.15.22:0/3054395472 >> v1:172.21.15.22:6790/0 conn(0x7fdca4032ce0 legacy=0x7fdca40c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.605+0000 7fdcabfff640 1 -- 172.21.15.22:0/3054395472 shutdown_connections 2024-03-29T11:37:07.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.605+0000 7fdcabfff640 1 -- 172.21.15.22:0/3054395472 >> 172.21.15.22:0/3054395472 conn(0x7fdca40058f0 msgr2=0x7fdca4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:07.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.605+0000 7fdcabfff640 1 -- 172.21.15.22:0/3054395472 shutdown_connections 2024-03-29T11:37:07.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.605+0000 7fdcabfff640 1 -- 172.21.15.22:0/3054395472 wait complete. 2024-03-29T11:37:07.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.606+0000 7fdcabfff640 10 monclient: build_initial_monmap 2024-03-29T11:37:07.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.606+0000 7fdcabfff640 10 monclient: monmap: 2024-03-29T11:37:07.618 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:07.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.607+0000 7fdcabfff640 1 librados: starting msgr at 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.607+0000 7fdcabfff640 1 librados: starting objecter 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.607+0000 7fdcabfff640 1 Processor -- start 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.607+0000 7fdcabfff640 1 -- start start 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.607+0000 7fdcabfff640 1 librados: setting wanted keys 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.607+0000 7fdcabfff640 1 librados: calling monclient init 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.607+0000 7fdcabfff640 10 monclient: init 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.608+0000 7fdcabfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.608+0000 7fdcabfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.608+0000 7fdcabfff640 10 monclient(hunting): picked mon.noname-c con 0x7fdca402a490 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:07.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.608+0000 7fdcabfff640 10 monclient(hunting): picked mon.noname-a con 0x7fdca402b810 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:07.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.609+0000 7fdcaaffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fdca402a490 0x7fdca416ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50924/0 (socket says 172.21.15.22:50924) 2024-03-29T11:37:07.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdcabfff640 10 monclient(hunting): picked mon.noname-b con 0x7fdca4032ce0 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:07.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdcabfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca416d440 con 0x7fdca402b810 2024-03-29T11:37:07.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdcabfff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca416d140 con 0x7fdca4032ce0 2024-03-29T11:37:07.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdcabfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca40c0230 con 0x7fdca402a490 2024-03-29T11:37:07.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdcabfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:07.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdcabfff640 10 monclient(hunting): authenticate will time out at 1077.887504s 2024-03-29T11:37:07.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdcaaffd640 1 -- 172.21.15.22:0/3049323813 learned_addr learned my addr 172.21.15.22:0/3049323813 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdc93fff640 10 client.?.objecter ms_handle_connect 0x7fdca4032ce0 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdc93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdc93fff640 10 client.?.objecter ms_handle_connect 0x7fdca402a490 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdc93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.611+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129566443 0 0) 0x7fdca416d140 con 0x7fdca4032ce0 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:07.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc98004eb0 con 0x7fdca4032ce0 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580414042 0 0) 0x7fdca40c0230 con 0x7fdca402a490 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca416d140 con 0x7fdca402a490 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 863480790 0 0) 0x7fdc98004eb0 con 0x7fdca4032ce0 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc98005490 con 0x7fdca4032ce0 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.612+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fdcac04f400 con 0x7fdca4032ce0 2024-03-29T11:37:07.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:07.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 monclient(hunting): dump: 2024-03-29T11:37:07.624 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:07.624 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:07.624 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.624 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.624 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.624 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4121569303 0 0) 0x7fdca416d140 con 0x7fdca402a490 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc98004eb0 con 0x7fdca402a490 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fdca0001920 con 0x7fdca402a490 2024-03-29T11:37:07.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 monclient(hunting): dump: 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:07.626 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 client.?.objecter ms_handle_connect 0x7fdca402b810 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.614+0000 7fdc93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702601084 0 0) 0x7fdca416d440 con 0x7fdca402b810 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:37:07.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca416d140 con 0x7fdca402b810 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4146801096 0 0) 0x7fdca416d140 con 0x7fdca402b810 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdca416d440 con 0x7fdca402b810 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fdc9c001510 con 0x7fdca402b810 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.615+0000 7fdc93fff640 10 monclient(hunting): dump: 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.628 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 534444166 0 0) 0x7fdc98005490 con 0x7fdca4032ce0 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 >> v1:172.21.15.22:6790/0 conn(0x7fdca402a490 legacy=0x7fdca416ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 >> v1:172.21.15.22:6789/0 conn(0x7fdca402b810 legacy=0x7fdca40be4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 1 monclient: found mon.b 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40c0510 con 0x7fdca4032ce0 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 10 monclient: _renew_subs 2024-03-29T11:37:07.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca40c3cb0 con 0x7fdca4032ce0 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 10 monclient: _renew_subs 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca40c4140 con 0x7fdca4032ce0 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdcabfff640 1 librados: init done 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcac04fdd0 con 0x7fdca4032ce0 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fdcac067bf0 con 0x7fdca4032ce0 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.616+0000 7fdc93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.617+0000 7fdc93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.617+0000 7fdc93fff640 10 monclient: dump: 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:07.631 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.618+0000 7fdcabfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.618+0000 7fdcabfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.618+0000 7fdcabfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:07.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.618+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca4034c70 con 0x7fdca4032ce0 2024-03-29T11:37:07.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.619+0000 7fdca97fa640 4 set_mon_vals no callback set 2024-03-29T11:37:07.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.620+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fdcac067d50 con 0x7fdca4032ce0 2024-03-29T11:37:07.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.621+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fdcac0ff510 con 0x7fdca4032ce0 2024-03-29T11:37:07.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.621+0000 7fdc93fff640 10 client.4321.objecter ms_dispatch 0x7fdcac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:07.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.621+0000 7fdc93fff640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:07.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.621+0000 7fdc93fff640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:07.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.621+0000 7fdc93fff640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-29T11:37:07.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.623+0000 7fdcaaffd640 10 monclient: get_auth_request con 0x7fdc9806e1c0 auth_method 0 2024-03-29T11:37:07.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.624+0000 7fdc93fff640 10 client.4321.objecter ms_handle_connect 0x7fdc9806e1c0 2024-03-29T11:37:07.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.628+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcac0cadf0 con 0x7fdca4032ce0 2024-03-29T11:37:07.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.628+0000 7fdc93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:07.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.628+0000 7fdc93fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:07.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.959+0000 7fdcabfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:07.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.959+0000 7fdcabfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:07.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.959+0000 7fdcabfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:07.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.959+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdca4004cd0 con 0x7fdca4032ce0 2024-03-29T11:37:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.963+0000 7fdc93fff640 1 -- 172.21.15.22:0/3049323813 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fdcac0ccd20 con 0x7fdca4032ce0 2024-03-29T11:37:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.963+0000 7fdc93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.963+0000 7fdc93fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:07.965 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:07.965 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:07.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.967+0000 7fdcabfff640 10 librados: watch_flush enter 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.967+0000 7fdcabfff640 10 librados: watch_flush exit 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.967+0000 7fdcabfff640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.967+0000 7fdcabfff640 10 client.4321.objecter successfully canceled tick 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 >> v1:172.21.15.204:6816/3848099522 conn(0x7fdc9806e1c0 legacy=0x7fdc98070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 10 monclient: shutdown 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 >> v1:172.21.15.204:6789/0 conn(0x7fdca4032ce0 legacy=0x7fdca40bebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 shutdown_connections 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 >> 172.21.15.22:0/3049323813 conn(0x7fdca40058f0 msgr2=0x7fdca40043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 shutdown_connections 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 1 -- 172.21.15.22:0/3049323813 wait complete. 2024-03-29T11:37:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:07.968+0000 7fdcabfff640 1 librados: shutdown 2024-03-29T11:37:07.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:14.790 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.942+0000 7f377f6f6640 10 monclient: get_monmap_and_config 2024-03-29T11:37:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.942+0000 7f377f6f6640 10 monclient: build_initial_monmap 2024-03-29T11:37:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.942+0000 7f377f6f6640 10 monclient: monmap: 2024-03-29T11:37:14.944 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:14.944 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:14.944 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:14.942604+0000 2024-03-29T11:37:14.944 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:14.942604+0000 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.945+0000 7f377f6f6640 1 Processor -- start 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.945+0000 7f377f6f6640 1 -- start start 2024-03-29T11:37:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.945+0000 7f377f6f6640 10 monclient: init 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 10 monclient(hunting): picked mon.noname-c con 0x7f37780ae250 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 10 monclient(hunting): picked mon.noname-b con 0x7f377813f970 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 10 monclient(hunting): picked mon.noname-a con 0x7f377811f1e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37780afe00 con 0x7f377811f1e0 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37780afb20 con 0x7f37780ae250 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37780aab80 con 0x7f377813f970 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377f6f6640 10 monclient(hunting): authenticate will time out at 1085.222561s 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377cc91640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f377813f970 0x7f37781420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40920/0 (socket says 172.21.15.22:40920) 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.946+0000 7f377cc91640 1 -- 172.21.15.22:0/3633222409 learned_addr learned my addr 172.21.15.22:0/3633222409 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.947+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323362483 0 0) 0x7f37780aab80 con 0x7f377813f970 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.947+0000 7f376ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.947+0000 7f376ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.947+0000 7f376ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.947+0000 7f376ffff640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.947+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3764003520 con 0x7f377813f970 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.947+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191442440 0 0) 0x7f3764003520 con 0x7f377813f970 2024-03-29T11:37:14.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.947+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 >> v1:172.21.15.204:6789/0 conn(0x7f37780ae250 legacy=0x7f377813f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 >> v1:172.21.15.22:6789/0 conn(0x7f377811f1e0 legacy=0x7f3778152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 1 monclient: found mon.noname-b 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37780aae60 con 0x7f377813f970 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 10 monclient: _finish_auth 0 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3768001690 con 0x7f377813f970 2024-03-29T11:37:14.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f376ffff640 10 monclient: dump: 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f377f6f6640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:37:14.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.951+0000 7f377f6f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.952+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3768001860 con 0x7f377813f970 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.952+0000 7f376ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.952+0000 7f376ffff640 1 -- 172.21.15.22:0/3633222409 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3768001c70 con 0x7f377813f970 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.952+0000 7f376ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.952+0000 7f376ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.952+0000 7f376ffff640 10 monclient: dump: 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:14.955 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.952+0000 7f377f6f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.952+0000 7f377f6f6640 4 set_mon_vals no callback set 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.953+0000 7f377f6f6640 10 monclient: shutdown 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.953+0000 7f377f6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.953+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633222409 >> v1:172.21.15.22:6790/0 conn(0x7f377813f970 legacy=0x7f37781420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.953+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633222409 shutdown_connections 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.953+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633222409 >> 172.21.15.22:0/3633222409 conn(0x7f37780b35e0 msgr2=0x7f37780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.954+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633222409 shutdown_connections 2024-03-29T11:37:14.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.954+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633222409 wait complete. 2024-03-29T11:37:14.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.955+0000 7f377f6f6640 10 monclient: build_initial_monmap 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.955+0000 7f377f6f6640 10 monclient: monmap: 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.955+0000 7f377f6f6640 1 librados: starting msgr at 2024-03-29T11:37:14.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.955+0000 7f377f6f6640 1 librados: starting objecter 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.955+0000 7f377f6f6640 1 Processor -- start 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.956+0000 7f377f6f6640 1 -- start start 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.956+0000 7f377f6f6640 1 librados: setting wanted keys 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.956+0000 7f377f6f6640 1 librados: calling monclient init 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.956+0000 7f377f6f6640 10 monclient: init 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 10 monclient(hunting): picked mon.noname-b con 0x7f37780ae250 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 10 monclient(hunting): picked mon.noname-c con 0x7f377813f970 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 10 monclient(hunting): picked mon.noname-a con 0x7f3778139bd0 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377813c2a0 con 0x7f3778139bd0 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377813a0f0 con 0x7f37780ae250 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377811bd00 con 0x7f377813f970 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:14.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.957+0000 7f377f6f6640 10 monclient(hunting): authenticate will time out at 1085.233194s 2024-03-29T11:37:14.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.959+0000 7f377cc91640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f377813f970 0x7f3778138b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42274/0 (socket says 172.21.15.22:42274) 2024-03-29T11:37:14.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.959+0000 7f377cc91640 1 -- 172.21.15.22:0/3633206946 learned_addr learned my addr 172.21.15.22:0/3633206946 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:14.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 10 client.?.objecter ms_handle_connect 0x7f377813f970 2024-03-29T11:37:14.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673650885 0 0) 0x7f377811bd00 con 0x7f377813f970 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375c004eb0 con 0x7f377813f970 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 591245276 0 0) 0x7f375c004eb0 con 0x7f377813f970 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f377811bd00 con 0x7f377813f970 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3768001c30 con 0x7f377813f970 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.960+0000 7f376dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.961+0000 7f376dffb640 10 monclient(hunting): dump: 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:14.962 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.961+0000 7f376dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.961+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2781499896 0 0) 0x7f377811bd00 con 0x7f377813f970 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.961+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 >> v1:172.21.15.204:6789/0 conn(0x7f37780ae250 legacy=0x7f377813a610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.961+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 >> v1:172.21.15.22:6789/0 conn(0x7f3778139bd0 legacy=0x7f37781e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.961+0000 7f376dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:14.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.961+0000 7f376dffb640 1 monclient: found mon.c 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37781dfa00 con 0x7f377813f970 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 client.?.objecter ms_handle_connect 0x7f3778139bd0 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3768004b00 con 0x7f377813f970 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3768004f50 con 0x7f377813f970 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.962+0000 7f376dffb640 10 monclient: dump: 2024-03-29T11:37:14.964 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:14.965 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.963+0000 7f376f7fe640 4 set_mon_vals no callback set 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 10 monclient: _renew_subs 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377813ad20 con 0x7f377813f970 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 10 monclient: _renew_subs 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:14.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3778139410 con 0x7f377813f970 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.964+0000 7f377f6f6640 1 librados: init done 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.966+0000 7f377f6f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.966+0000 7f377f6f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.966+0000 7f377f6f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.966+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377813a280 con 0x7f377813f970 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.967+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f37680050b0 con 0x7f377813f970 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.968+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f3768084c00 con 0x7f377813f970 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.968+0000 7f376dffb640 10 client.4325.objecter ms_dispatch 0x7f3778000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.968+0000 7f376dffb640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.968+0000 7f376dffb640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.968+0000 7f377d492640 10 monclient: get_auth_request con 0x7f375c06dc80 auth_method 0 2024-03-29T11:37:14.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.968+0000 7f376dffb640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:37:14.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.969+0000 7f376dffb640 10 client.4325.objecter ms_handle_connect 0x7f375c06dc80 2024-03-29T11:37:14.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.976+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37680504e0 con 0x7f377813f970 2024-03-29T11:37:14.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.976+0000 7f376dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:14.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:14.976+0000 7f376dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:15.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.300+0000 7f377f6f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.300+0000 7f377f6f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.300+0000 7f377f6f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:15.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.300+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f377813aeb0 con 0x7f377813f970 2024-03-29T11:37:15.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.301+0000 7f376dffb640 1 -- 172.21.15.22:0/3633206946 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f3768052410 con 0x7f377813f970 2024-03-29T11:37:15.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.301+0000 7f376dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.301+0000 7f376dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:15.301 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:15.301 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:15.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.305+0000 7f377f6f6640 10 librados: watch_flush enter 2024-03-29T11:37:15.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.305+0000 7f377f6f6640 10 librados: watch_flush exit 2024-03-29T11:37:15.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.305+0000 7f377f6f6640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:37:15.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.305+0000 7f377f6f6640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 >> v1:172.21.15.204:6816/3848099522 conn(0x7f375c06dc80 legacy=0x7f375c070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 10 monclient: shutdown 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 >> v1:172.21.15.22:6790/0 conn(0x7f377813f970 legacy=0x7f3778138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 shutdown_connections 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 >> 172.21.15.22:0/3633206946 conn(0x7f37780b35e0 msgr2=0x7f37780aa060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 shutdown_connections 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 1 -- 172.21.15.22:0/3633206946 wait complete. 2024-03-29T11:37:15.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:15.306+0000 7f377f6f6640 1 librados: shutdown 2024-03-29T11:37:15.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:22.132 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:22.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.284+0000 7f024c457640 10 monclient: get_monmap_and_config 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.284+0000 7f024c457640 10 monclient: build_initial_monmap 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.284+0000 7f024c457640 10 monclient: monmap: 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:22.285984+0000 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:22.285984+0000 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:22.285 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:22.286 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:22.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.287+0000 7f024c457640 1 Processor -- start 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.287+0000 7f024c457640 1 -- start start 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.287+0000 7f024c457640 10 monclient: init 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.288+0000 7f024c457640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.288+0000 7f024c457640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.288+0000 7f024c457640 10 monclient(hunting): picked mon.noname-a con 0x7f02440b2520 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:22.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.288+0000 7f024c457640 10 monclient(hunting): picked mon.noname-c con 0x7f02440b2900 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.288+0000 7f024c457640 10 monclient(hunting): picked mon.noname-b con 0x7f02440a9c70 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.288+0000 7f024c457640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0244142170 con 0x7f02440b2520 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.288+0000 7f024c457640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02440b2260 con 0x7f02440b2900 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.289+0000 7f024c457640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02440aaf60 con 0x7f02440a9c70 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.289+0000 7f024c457640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:22.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.289+0000 7f024c457640 10 monclient(hunting): authenticate will time out at 1092.566022s 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.296+0000 7f024a9f4640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f02440a9c70 0x7f024414d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:46577/0 (socket says 172.21.15.22:46577) 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.296+0000 7f024a9f4640 1 -- 172.21.15.22:0/4009606139 learned_addr learned my addr 172.21.15.22:0/4009606139 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.296+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626618042 0 0) 0x7f02440aaf60 con 0x7f02440a9c70 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.296+0000 7f02491f1640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.296+0000 7f02491f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.296+0000 7f02491f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.296+0000 7f02491f1640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:37:22.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.296+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0230003520 con 0x7f02440a9c70 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557730077 0 0) 0x7f0244142170 con 0x7f02440b2520 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02440aaf60 con 0x7f02440b2520 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160856783 0 0) 0x7f0230003520 con 0x7f02440a9c70 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 >> v1:172.21.15.204:6789/0 conn(0x7f02440b2900 legacy=0x7f0244142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 >> v1:172.21.15.22:6789/0 conn(0x7f02440b2520 legacy=0x7f02440b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient: _finish_hunting 0 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 1 monclient: found mon.noname-b 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:22.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244142d10 con 0x7f02440a9c70 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient: _finish_auth 0 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient: _check_auth_tickets 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f023c001a40 con 0x7f02440a9c70 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f02491f1640 10 monclient: dump: 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.301 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:22.302 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:22.302 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:22.302 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:22.302 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:22.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f024c457640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:37:22.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.298+0000 7f024c457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:22.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.299+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f023c001ba0 con 0x7f02440a9c70 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.299+0000 7f02491f1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.299+0000 7f02491f1640 1 -- 172.21.15.22:0/4009606139 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f023c001f70 con 0x7f02440a9c70 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.299+0000 7f02491f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.299+0000 7f02491f1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.299+0000 7f02491f1640 10 monclient: dump: 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:22.304 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.299+0000 7f024c457640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.299+0000 7f024c457640 4 set_mon_vals no callback set 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.300+0000 7f024c457640 10 monclient: shutdown 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.300+0000 7f024c457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.300+0000 7f024c457640 1 -- 172.21.15.22:0/4009606139 >> v1:172.21.15.22:6790/0 conn(0x7f02440a9c70 legacy=0x7f024414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.301+0000 7f024c457640 1 -- 172.21.15.22:0/4009606139 shutdown_connections 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.301+0000 7f024c457640 1 -- 172.21.15.22:0/4009606139 >> 172.21.15.22:0/4009606139 conn(0x7f024406d180 msgr2=0x7f0244066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:22.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.305+0000 7f024c457640 1 -- 172.21.15.22:0/4009606139 shutdown_connections 2024-03-29T11:37:22.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.305+0000 7f024c457640 1 -- 172.21.15.22:0/4009606139 wait complete. 2024-03-29T11:37:22.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.306+0000 7f024c457640 10 monclient: build_initial_monmap 2024-03-29T11:37:22.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.306+0000 7f024c457640 10 monclient: monmap: 2024-03-29T11:37:22.306 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:22.307 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:22.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.307+0000 7f024c457640 1 librados: starting msgr at 2024-03-29T11:37:22.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.307+0000 7f024c457640 1 librados: starting objecter 2024-03-29T11:37:22.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.307+0000 7f024c457640 1 Processor -- start 2024-03-29T11:37:22.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.307+0000 7f024c457640 1 -- start start 2024-03-29T11:37:22.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.307+0000 7f024c457640 1 librados: setting wanted keys 2024-03-29T11:37:22.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.307+0000 7f024c457640 1 librados: calling monclient init 2024-03-29T11:37:22.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.307+0000 7f024c457640 10 monclient: init 2024-03-29T11:37:22.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.308+0000 7f024c457640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:22.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.308+0000 7f024c457640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:22.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.311+0000 7f024c457640 10 monclient(hunting): picked mon.noname-a con 0x7f02440a9c70 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:22.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.311+0000 7f024a1f3640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f02440a9c70 0x7f024413baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:42433/0 (socket says 172.21.15.22:42433) 2024-03-29T11:37:22.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.312+0000 7f024c457640 10 monclient(hunting): picked mon.noname-c con 0x7f02440b2520 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:22.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.312+0000 7f024c457640 10 monclient(hunting): picked mon.noname-b con 0x7f024406de70 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:22.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.312+0000 7f024c457640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024413b0e0 con 0x7f02440a9c70 2024-03-29T11:37:22.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.312+0000 7f024c457640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024406f260 con 0x7f024406de70 2024-03-29T11:37:22.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.312+0000 7f024c457640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02441384a0 con 0x7f02440b2520 2024-03-29T11:37:22.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.312+0000 7f024c457640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:22.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.312+0000 7f024c457640 10 monclient(hunting): authenticate will time out at 1092.589134s 2024-03-29T11:37:22.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.314+0000 7f024a1f3640 1 -- 172.21.15.22:0/526221391 learned_addr learned my addr 172.21.15.22:0/526221391 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:22.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 client.?.objecter ms_handle_connect 0x7f02440a9c70 2024-03-29T11:37:22.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:22.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 client.?.objecter ms_handle_connect 0x7f02440b2520 2024-03-29T11:37:22.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:22.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599284483 0 0) 0x7f024413b0e0 con 0x7f02440a9c70 2024-03-29T11:37:22.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0228004eb0 con 0x7f02440a9c70 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278682339 0 0) 0x7f02441384a0 con 0x7f02440b2520 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.315+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024413b0e0 con 0x7f02440b2520 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2025447628 0 0) 0x7f024413b0e0 con 0x7f02440b2520 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02441384a0 con 0x7f02440b2520 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f0234001640 con 0x7f02440b2520 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 10 monclient(hunting): dump: 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2224601495 0 0) 0x7f02441384a0 con 0x7f02440b2520 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.316+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 >> v1:172.21.15.204:6789/0 conn(0x7f024406de70 legacy=0x7f02441e64a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:22.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 >> v1:172.21.15.22:6789/0 conn(0x7f02440a9c70 legacy=0x7f024413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:22.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 1 monclient: found mon.c 2024-03-29T11:37:22.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:22.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:22.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244138630 con 0x7f02440b2520 2024-03-29T11:37:22.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 10 monclient: _finish_auth 0 2024-03-29T11:37:22.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:22.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:22.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02340017a0 con 0x7f02440b2520 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f023affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.318+0000 7f02489f0640 4 set_mon_vals no callback set 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.319+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f02340035f0 con 0x7f02440b2520 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.319+0000 7f023affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.319+0000 7f023affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.319+0000 7f023affd640 10 monclient: dump: 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:22.324 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 10 monclient: _renew_subs 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0244135b90 con 0x7f02440b2520 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 10 monclient: _renew_subs 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02441e7bc0 con 0x7f02440b2520 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.320+0000 7f024c457640 1 librados: init done 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.322+0000 7f024c457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.322+0000 7f024c457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.322+0000 7f024c457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:22.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.322+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024406f060 con 0x7f02440b2520 2024-03-29T11:37:22.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.323+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f02340195c0 con 0x7f02440b2520 2024-03-29T11:37:22.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.323+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f0234083610 con 0x7f02440b2520 2024-03-29T11:37:22.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.323+0000 7f023affd640 10 client.4331.objecter ms_dispatch 0x7f0244000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:22.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.323+0000 7f023affd640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:22.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.323+0000 7f023affd640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:22.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.324+0000 7f023affd640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:37:22.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.324+0000 7f024a1f3640 10 monclient: get_auth_request con 0x7f022806dd20 auth_method 0 2024-03-29T11:37:22.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.327+0000 7f023affd640 10 client.4331.objecter ms_handle_connect 0x7f022806dd20 2024-03-29T11:37:22.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.335+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f023404ed20 con 0x7f02440b2520 2024-03-29T11:37:22.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.335+0000 7f023affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.335+0000 7f023affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:22.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.664+0000 7f024c457640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:22.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.664+0000 7f024c457640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:22.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.664+0000 7f024c457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:37:22.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.664+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0244135d20 con 0x7f02440b2520 2024-03-29T11:37:22.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.667+0000 7f023affd640 1 -- 172.21.15.22:0/526221391 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f0234050e20 con 0x7f02440b2520 2024-03-29T11:37:22.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.667+0000 7f023affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:22.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.667+0000 7f023affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:22.667 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:22.667 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.670+0000 7f024c457640 10 librados: watch_flush enter 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 10 librados: watch_flush exit 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 >> v1:172.21.15.204:6816/3848099522 conn(0x7f022806dd20 legacy=0x7f02280701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 10 monclient: shutdown 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 >> v1:172.21.15.22:6790/0 conn(0x7f02440b2520 legacy=0x7f0244139ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 shutdown_connections 2024-03-29T11:37:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 >> 172.21.15.22:0/526221391 conn(0x7f024406d180 msgr2=0x7f0244140240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 shutdown_connections 2024-03-29T11:37:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 1 -- 172.21.15.22:0/526221391 wait complete. 2024-03-29T11:37:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:22.671+0000 7f024c457640 1 librados: shutdown 2024-03-29T11:37:22.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:29.499 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:29.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.646+0000 7f321d34b640 10 monclient: get_monmap_and_config 2024-03-29T11:37:29.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.646+0000 7f321d34b640 10 monclient: build_initial_monmap 2024-03-29T11:37:29.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.646+0000 7f321d34b640 10 monclient: monmap: 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:29.647896+0000 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:29.647896+0000 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:29.647 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:29.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.650+0000 7f321d34b640 1 Processor -- start 2024-03-29T11:37:29.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.651+0000 7f321d34b640 1 -- start start 2024-03-29T11:37:29.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.651+0000 7f321d34b640 10 monclient: init 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.651+0000 7f321d34b640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.651+0000 7f321d34b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.652+0000 7f321d34b640 10 monclient(hunting): picked mon.noname-a con 0x7f32181457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.652+0000 7f321d34b640 10 monclient(hunting): picked mon.noname-b con 0x7f321813f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.652+0000 7f321d34b640 10 monclient(hunting): picked mon.noname-c con 0x7f32180a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.652+0000 7f321d34b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32180aaef0 con 0x7f32181457f0 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.652+0000 7f321d34b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3218145bd0 con 0x7f32180a9d70 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.652+0000 7f321d34b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3218145eb0 con 0x7f321813f650 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.652+0000 7f321d34b640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:29.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.652+0000 7f321d34b640 10 monclient(hunting): authenticate will time out at 1099.929106s 2024-03-29T11:37:29.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.656+0000 7f32177fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f32180a9d70 0x7f321814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:59701/0 (socket says 172.21.15.22:59701) 2024-03-29T11:37:29.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.656+0000 7f32177fe640 1 -- 172.21.15.22:0/2398189707 learned_addr learned my addr 172.21.15.22:0/2398189707 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:29.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.659+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692857412 0 0) 0x7f3218145bd0 con 0x7f32180a9d70 2024-03-29T11:37:29.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.659+0000 7f3215ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:29.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.659+0000 7f3215ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:29.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.659+0000 7f3215ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:29.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.659+0000 7f3215ffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:37:29.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.659+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3200003520 con 0x7f32180a9d70 2024-03-29T11:37:29.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306545740 0 0) 0x7f3200003520 con 0x7f32180a9d70 2024-03-29T11:37:29.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 >> v1:172.21.15.22:6790/0 conn(0x7f321813f650 legacy=0x7f32180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 >> v1:172.21.15.22:6789/0 conn(0x7f32181457f0 legacy=0x7f321813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:29.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:29.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 1 monclient: found mon.noname-c 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3218146340 con 0x7f32180a9d70 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3210001a10 con 0x7f32180a9d70 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f3215ffb640 10 monclient: dump: 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:29.663 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f321d34b640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.661+0000 7f321d34b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.662+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3210001be0 con 0x7f32180a9d70 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.662+0000 7f3215ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.662+0000 7f3215ffb640 1 -- 172.21.15.22:0/2398189707 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3210002030 con 0x7f32180a9d70 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.662+0000 7f3215ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.662+0000 7f3215ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:29.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.662+0000 7f3215ffb640 10 monclient: dump: 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.662+0000 7f321d34b640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:29.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.662+0000 7f321d34b640 4 set_mon_vals no callback set 2024-03-29T11:37:29.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.663+0000 7f321d34b640 10 monclient: shutdown 2024-03-29T11:37:29.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.663+0000 7f321d34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:29.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.663+0000 7f321d34b640 1 -- 172.21.15.22:0/2398189707 >> v1:172.21.15.204:6789/0 conn(0x7f32180a9d70 legacy=0x7f321814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.665+0000 7f321d34b640 1 -- 172.21.15.22:0/2398189707 shutdown_connections 2024-03-29T11:37:29.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.665+0000 7f321d34b640 1 -- 172.21.15.22:0/2398189707 >> 172.21.15.22:0/2398189707 conn(0x7f3218066a00 msgr2=0x7f3218066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:29.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.666+0000 7f321d34b640 1 -- 172.21.15.22:0/2398189707 shutdown_connections 2024-03-29T11:37:29.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.666+0000 7f321d34b640 1 -- 172.21.15.22:0/2398189707 wait complete. 2024-03-29T11:37:29.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.667+0000 7f321d34b640 10 monclient: build_initial_monmap 2024-03-29T11:37:29.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.667+0000 7f321d34b640 10 monclient: monmap: 2024-03-29T11:37:29.668 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:29.668 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.667+0000 7f321d34b640 1 librados: starting msgr at 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.667+0000 7f321d34b640 1 librados: starting objecter 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.667+0000 7f321d34b640 1 Processor -- start 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 1 -- start start 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 1 librados: setting wanted keys 2024-03-29T11:37:29.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 1 librados: calling monclient init 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 10 monclient: init 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 10 monclient(hunting): picked mon.noname-c con 0x7f32180a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 10 monclient(hunting): picked mon.noname-b con 0x7f321813f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.668+0000 7f321d34b640 10 monclient(hunting): picked mon.noname-a con 0x7f32181457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.669+0000 7f321d34b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321813aa30 con 0x7f32181457f0 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.669+0000 7f321d34b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321813ad10 con 0x7f321813f650 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.669+0000 7f321d34b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3218142ce0 con 0x7f32180a9d70 2024-03-29T11:37:29.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.669+0000 7f321d34b640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:29.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.669+0000 7f321d34b640 10 monclient(hunting): authenticate will time out at 1099.946126s 2024-03-29T11:37:29.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.672+0000 7f32167fc640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f321813f650 0x7f3218138f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:59827/0 (socket says 172.21.15.22:59827) 2024-03-29T11:37:29.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.672+0000 7f32167fc640 1 -- 172.21.15.22:0/4146552107 learned_addr learned my addr 172.21.15.22:0/4146552107 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:29.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.673+0000 7f31ff7fe640 10 client.?.objecter ms_handle_connect 0x7f321813f650 2024-03-29T11:37:29.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.673+0000 7f31ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.673+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072535364 0 0) 0x7f321813ad10 con 0x7f321813f650 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.673+0000 7f31ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.673+0000 7f31ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.673+0000 7f31ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.673+0000 7f31ff7fe640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.673+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3204004eb0 con 0x7f321813f650 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.674+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660987793 0 0) 0x7f3204004eb0 con 0x7f321813f650 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.674+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f321813ad10 con 0x7f321813f650 2024-03-29T11:37:29.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.674+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f320c001a30 con 0x7f321813f650 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.674+0000 7f31ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.674+0000 7f31ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.674+0000 7f31ff7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:29.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.674+0000 7f31ff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:29.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 client.?.objecter ms_handle_connect 0x7f32181457f0 2024-03-29T11:37:29.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:29.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335705454 0 0) 0x7f321813aa30 con 0x7f32181457f0 2024-03-29T11:37:29.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:29.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:29.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:29.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3204004eb0 con 0x7f32181457f0 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3817561576 0 0) 0x7f321813ad10 con 0x7f321813f650 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 >> v1:172.21.15.22:6790/0 conn(0x7f32180a9d70 legacy=0x7f321813b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 >> v1:172.21.15.22:6789/0 conn(0x7f32181457f0 legacy=0x7f32181e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 1 monclient: found mon.b 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32181429b0 con 0x7f321813f650 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.677+0000 7f31ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:37:29.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 10 monclient: _renew_subs 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321813bb20 con 0x7f321813f650 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 10 monclient: _renew_subs 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32181397f0 con 0x7f321813f650 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.678+0000 7f321d34b640 1 librados: init done 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.679+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f320c001d30 con 0x7f321813f650 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.679+0000 7f31ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.679+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f320c003a60 con 0x7f321813f650 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.679+0000 7f31ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.679+0000 7f31ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.679+0000 7f31ff7fe640 10 monclient: dump: 2024-03-29T11:37:29.683 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.679+0000 7f3214ff9640 4 set_mon_vals no callback set 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.680+0000 7f321d34b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.680+0000 7f321d34b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.680+0000 7f321d34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:29.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.680+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f8005080 con 0x7f321813f650 2024-03-29T11:37:29.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.686+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f320c003bc0 con 0x7f321813f650 2024-03-29T11:37:29.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.687+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f320c084b80 con 0x7f321813f650 2024-03-29T11:37:29.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.687+0000 7f31ff7fe640 10 client.4327.objecter ms_dispatch 0x7f3218000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:29.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.687+0000 7f31ff7fe640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:29.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.687+0000 7f31ff7fe640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:29.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.687+0000 7f31ff7fe640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T11:37:29.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.687+0000 7f3216ffd640 10 monclient: get_auth_request con 0x7f320406dd30 auth_method 0 2024-03-29T11:37:29.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.689+0000 7f31ff7fe640 10 client.4327.objecter ms_handle_connect 0x7f320406dd30 2024-03-29T11:37:29.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.690+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f320c050290 con 0x7f321813f650 2024-03-29T11:37:29.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.690+0000 7f31ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:29.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:29.690+0000 7f31ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:30.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.009+0000 7f321d34b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.009+0000 7f321d34b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.009+0000 7f321d34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:30.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.009+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31f8005330 con 0x7f321813f650 2024-03-29T11:37:30.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.010+0000 7f31ff7fe640 1 -- 172.21.15.22:0/4146552107 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f320c052390 con 0x7f321813f650 2024-03-29T11:37:30.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.010+0000 7f31ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.010+0000 7f31ff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:30.013 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:30.013 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 10 librados: watch_flush enter 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 10 librados: watch_flush exit 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 10 client.4327.objecter successfully canceled tick 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 >> v1:172.21.15.204:6816/3848099522 conn(0x7f320406dd30 legacy=0x7f32040701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 10 monclient: shutdown 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 >> v1:172.21.15.204:6789/0 conn(0x7f321813f650 legacy=0x7f3218138f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 shutdown_connections 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.017+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 >> 172.21.15.22:0/4146552107 conn(0x7f3218066a00 msgr2=0x7f32180b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:30.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.018+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 shutdown_connections 2024-03-29T11:37:30.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.018+0000 7f321d34b640 1 -- 172.21.15.22:0/4146552107 wait complete. 2024-03-29T11:37:30.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:30.018+0000 7f321d34b640 1 librados: shutdown 2024-03-29T11:37:30.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:36.842 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:36.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.997+0000 7fe551505640 10 monclient: get_monmap_and_config 2024-03-29T11:37:36.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.997+0000 7fe551505640 10 monclient: build_initial_monmap 2024-03-29T11:37:36.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.998+0000 7fe551505640 10 monclient: monmap: 2024-03-29T11:37:36.998 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:36.998 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:36.998 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:36.999277+0000 2024-03-29T11:37:36.999 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:36.999277+0000 2024-03-29T11:37:36.999 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:36.999 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:36.999 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:36.999 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:36.999 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:36.999 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:37.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.998+0000 7fe551505640 1 Processor -- start 2024-03-29T11:37:37.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.999+0000 7fe551505640 1 -- start start 2024-03-29T11:37:37.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.999+0000 7fe551505640 10 monclient: init 2024-03-29T11:37:37.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.999+0000 7fe551505640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.999+0000 7fe551505640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:36.999+0000 7fe551505640 10 monclient(hunting): picked mon.noname-b con 0x7fe54c0b2800 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.000+0000 7fe551505640 10 monclient(hunting): picked mon.noname-a con 0x7fe54c0a9c00 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.000+0000 7fe551505640 10 monclient(hunting): picked mon.noname-c con 0x7fe54c0b38c0 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.000+0000 7fe551505640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c141eb0 con 0x7fe54c0a9c00 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.000+0000 7fe551505640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c0b22d0 con 0x7fe54c0b38c0 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.000+0000 7fe551505640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c142d30 con 0x7fe54c0b2800 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.000+0000 7fe551505640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:37.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.000+0000 7fe551505640 10 monclient(hunting): authenticate will time out at 1107.277843s 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.002+0000 7fe54bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe54c0b2800 0x7fe54c142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56115/0 (socket says 172.21.15.22:56115) 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.002+0000 7fe54bfff640 1 -- 172.21.15.22:0/1226341664 learned_addr learned my addr 172.21.15.22:0/1226341664 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235698735 0 0) 0x7fe54c142d30 con 0x7fe54c0b2800 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534003520 con 0x7fe54c0b2800 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856052995 0 0) 0x7fe534003520 con 0x7fe54c0b2800 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 >> v1:172.21.15.204:6789/0 conn(0x7fe54c0b38c0 legacy=0x7fe54c14d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 >> v1:172.21.15.22:6789/0 conn(0x7fe54c0a9c00 legacy=0x7fe54c13bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:37.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 1 monclient: found mon.noname-b 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c0aaf90 con 0x7fe54c0b2800 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient: _finish_auth 0 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe53c001690 con 0x7fe54c0b2800 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.003+0000 7fe54affd640 10 monclient: dump: 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.007 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe551505640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe551505640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe53c001930 con 0x7fe54c0b2800 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe54affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe54affd640 1 -- 172.21.15.22:0/1226341664 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe53c001cc0 con 0x7fe54c0b2800 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe54affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe54affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe54affd640 10 monclient: dump: 2024-03-29T11:37:37.008 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe551505640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.004+0000 7fe551505640 4 set_mon_vals no callback set 2024-03-29T11:37:37.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.005+0000 7fe551505640 10 monclient: shutdown 2024-03-29T11:37:37.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.005+0000 7fe551505640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:37.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.005+0000 7fe551505640 1 -- 172.21.15.22:0/1226341664 >> v1:172.21.15.22:6790/0 conn(0x7fe54c0b2800 legacy=0x7fe54c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.007+0000 7fe551505640 1 -- 172.21.15.22:0/1226341664 shutdown_connections 2024-03-29T11:37:37.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.007+0000 7fe551505640 1 -- 172.21.15.22:0/1226341664 >> 172.21.15.22:0/1226341664 conn(0x7fe54c066a00 msgr2=0x7fe54c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:37.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.008+0000 7fe551505640 1 -- 172.21.15.22:0/1226341664 shutdown_connections 2024-03-29T11:37:37.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.008+0000 7fe551505640 1 -- 172.21.15.22:0/1226341664 wait complete. 2024-03-29T11:37:37.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.009+0000 7fe551505640 10 monclient: build_initial_monmap 2024-03-29T11:37:37.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.009+0000 7fe551505640 10 monclient: monmap: 2024-03-29T11:37:37.011 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:37.012 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:37.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.010+0000 7fe551505640 1 librados: starting msgr at 2024-03-29T11:37:37.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.010+0000 7fe551505640 1 librados: starting objecter 2024-03-29T11:37:37.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.010+0000 7fe551505640 1 Processor -- start 2024-03-29T11:37:37.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.010+0000 7fe551505640 1 -- start start 2024-03-29T11:37:37.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.010+0000 7fe551505640 1 librados: setting wanted keys 2024-03-29T11:37:37.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.011+0000 7fe551505640 1 librados: calling monclient init 2024-03-29T11:37:37.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.011+0000 7fe551505640 10 monclient: init 2024-03-29T11:37:37.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.012+0000 7fe551505640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:37.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.012+0000 7fe551505640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:37.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.013+0000 7fe551505640 10 monclient(hunting): picked mon.noname-a con 0x7fe54c0a9c00 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:37.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe551505640 10 monclient(hunting): picked mon.noname-b con 0x7fe54c0b2800 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:37.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe551505640 10 monclient(hunting): picked mon.noname-c con 0x7fe54c0b38c0 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:37.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe551505640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c0ac390 con 0x7fe54c0a9c00 2024-03-29T11:37:37.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe551505640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c070b40 con 0x7fe54c0b2800 2024-03-29T11:37:37.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe551505640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c13bfa0 con 0x7fe54c0b38c0 2024-03-29T11:37:37.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe551505640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:37.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe551505640 10 monclient(hunting): authenticate will time out at 1107.293763s 2024-03-29T11:37:37.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe54bfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fe54c0a9c00 0x7fe54c13b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46097/0 (socket says 172.21.15.22:46097) 2024-03-29T11:37:37.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe54bfff640 1 -- 172.21.15.22:0/1367308294 learned_addr learned my addr 172.21.15.22:0/1367308294 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.016+0000 7fe548ff9640 10 client.?.objecter ms_handle_connect 0x7fe54c0a9c00 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780209840 0 0) 0x7fe54c0ac390 con 0x7fe54c0a9c00 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe538004eb0 con 0x7fe54c0a9c00 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 10 client.?.objecter ms_handle_connect 0x7fe54c0b2800 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 219083922 0 0) 0x7fe538004eb0 con 0x7fe54c0a9c00 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe54c0ac390 con 0x7fe54c0a9c00 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe53c001650 con 0x7fe54c0a9c00 2024-03-29T11:37:37.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.017+0000 7fe548ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:37.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:37.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655200405 0 0) 0x7fe54c070b40 con 0x7fe54c0b2800 2024-03-29T11:37:37.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:37:37.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.018+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe538004eb0 con 0x7fe54c0b2800 2024-03-29T11:37:37.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 10 client.?.objecter ms_handle_connect 0x7fe54c0b38c0 2024-03-29T11:37:37.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2572928165 0 0) 0x7fe538004eb0 con 0x7fe54c0b2800 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe54c070b40 con 0x7fe54c0b2800 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe540001510 con 0x7fe54c0b2800 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:37.026 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 809562452 0 0) 0x7fe54c0ac390 con 0x7fe54c0a9c00 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 >> v1:172.21.15.22:6790/0 conn(0x7fe54c0b38c0 legacy=0x7fe54c1de1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 >> v1:172.21.15.204:6789/0 conn(0x7fe54c0b2800 legacy=0x7fe54c0b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.021+0000 7fe548ff9640 1 monclient: found mon.a 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe548ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe548ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c13aab0 con 0x7fe54c0a9c00 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe548ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe548ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe548ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:37:37.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 10 monclient: _renew_subs 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c1d6b40 con 0x7fe54c0a9c00 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 10 monclient: _renew_subs 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c13bc60 con 0x7fe54c0a9c00 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.022+0000 7fe551505640 1 librados: init done 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.024+0000 7fe5327fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.024+0000 7fe5327fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.024+0000 7fe5327fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:37.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.024+0000 7fe5327fc640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c1d6d80 con 0x7fe54c0a9c00 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.029+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe53c004c60 con 0x7fe54c0a9c00 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.029+0000 7fe548ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.029+0000 7fe54a7fc640 4 set_mon_vals no callback set 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.030+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe53c005030 con 0x7fe54c0a9c00 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.030+0000 7fe548ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.030+0000 7fe548ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.030+0000 7fe548ff9640 10 monclient: dump: 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.031 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:37.032 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:37.032 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:37.032 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:37.032 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:37.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.030+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fe53c005250 con 0x7fe54c0a9c00 2024-03-29T11:37:37.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.031+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fe53c084dd0 con 0x7fe54c0a9c00 2024-03-29T11:37:37.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.031+0000 7fe548ff9640 10 client.4344.objecter ms_dispatch 0x7fe54c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:37.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.031+0000 7fe548ff9640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:37.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.031+0000 7fe548ff9640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:37.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.031+0000 7fe548ff9640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T11:37:37.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.033+0000 7fe54b7fe640 10 monclient: get_auth_request con 0x7fe53806de00 auth_method 0 2024-03-29T11:37:37.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.033+0000 7fe548ff9640 10 client.4344.objecter ms_handle_connect 0x7fe53806de00 2024-03-29T11:37:37.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.035+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe53c050730 con 0x7fe54c0a9c00 2024-03-29T11:37:37.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.036+0000 7fe548ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.036+0000 7fe548ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:37.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.358+0000 7fe5327fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.358+0000 7fe5327fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.358+0000 7fe5327fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:37.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.358+0000 7fe5327fc640 1 -- 172.21.15.22:0/1367308294 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe54c04f730 con 0x7fe54c0a9c00 2024-03-29T11:37:37.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.362+0000 7fe548ff9640 1 -- 172.21.15.22:0/1367308294 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fe53c052660 con 0x7fe54c0a9c00 2024-03-29T11:37:37.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.362+0000 7fe548ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.362+0000 7fe548ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:37.362 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:37.362 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 10 librados: watch_flush enter 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 10 librados: watch_flush exit 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 10 client.4344.objecter successfully canceled tick 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 1 -- 172.21.15.22:0/1367308294 >> v1:172.21.15.204:6816/3848099522 conn(0x7fe53806de00 legacy=0x7fe5380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 10 monclient: shutdown 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 1 -- 172.21.15.22:0/1367308294 >> v1:172.21.15.22:6789/0 conn(0x7fe54c0a9c00 legacy=0x7fe54c13b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 1 -- 172.21.15.22:0/1367308294 shutdown_connections 2024-03-29T11:37:37.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.365+0000 7fe5327fc640 1 -- 172.21.15.22:0/1367308294 >> 172.21.15.22:0/1367308294 conn(0x7fe54c066a00 msgr2=0x7fe54c140370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:37.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.366+0000 7fe5327fc640 1 -- 172.21.15.22:0/1367308294 shutdown_connections 2024-03-29T11:37:37.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.366+0000 7fe5327fc640 1 -- 172.21.15.22:0/1367308294 wait complete. 2024-03-29T11:37:37.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:37.366+0000 7fe5327fc640 1 librados: shutdown 2024-03-29T11:37:37.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:44.191 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:44.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.350+0000 7fe6caa55640 10 monclient: get_monmap_and_config 2024-03-29T11:37:44.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.350+0000 7fe6caa55640 10 monclient: build_initial_monmap 2024-03-29T11:37:44.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.350+0000 7fe6caa55640 10 monclient: monmap: 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:44.351404+0000 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:44.351404+0000 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:44.352 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:44.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.354+0000 7fe6caa55640 1 Processor -- start 2024-03-29T11:37:44.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.354+0000 7fe6caa55640 1 -- start start 2024-03-29T11:37:44.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.354+0000 7fe6caa55640 10 monclient: init 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c40a9930 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c4149dc0 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c414ff40 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c40aae60 con 0x7fe6c40a9930 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c40b4310 con 0x7fe6c4149dc0 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c40afdd0 con 0x7fe6c414ff40 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.355+0000 7fe6caa55640 10 monclient(hunting): authenticate will time out at 1114.632387s 2024-03-29T11:37:44.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.358+0000 7fe6c37fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fe6c4149dc0 0x7fe6c414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:37833/0 (socket says 172.21.15.22:37833) 2024-03-29T11:37:44.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.358+0000 7fe6c37fe640 1 -- 172.21.15.22:0/2651617866 learned_addr learned my addr 172.21.15.22:0/2651617866 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:44.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.359+0000 7fe6c2ffd640 1 -- 172.21.15.22:0/2651617866 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696869781 0 0) 0x7fe6c40b4310 con 0x7fe6c4149dc0 2024-03-29T11:37:44.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.359+0000 7fe6c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.359+0000 7fe6c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.359+0000 7fe6c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.359+0000 7fe6c2ffd640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:37:44.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.359+0000 7fe6c2ffd640 1 -- 172.21.15.22:0/2651617866 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ac003520 con 0x7fe6c4149dc0 2024-03-29T11:37:44.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 1 -- 172.21.15.22:0/2651617866 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3678756788 0 0) 0x7fe6ac003520 con 0x7fe6c4149dc0 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 1 -- 172.21.15.22:0/2651617866 >> v1:172.21.15.22:6790/0 conn(0x7fe6c414ff40 legacy=0x7fe6c4152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 1 -- 172.21.15.22:0/2651617866 >> v1:172.21.15.22:6789/0 conn(0x7fe6c40a9930 legacy=0x7fe6c4143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 1 -- 172.21.15.22:0/2651617866 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c414a950 con 0x7fe6c4149dc0 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6caa55640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6caa55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:44.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 1 -- 172.21.15.22:0/2651617866 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe6bc001a10 con 0x7fe6c4149dc0 2024-03-29T11:37:44.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:44.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.360+0000 7fe6c2ffd640 10 monclient: dump: 2024-03-29T11:37:44.364 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:44.364 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.361+0000 7fe6caa55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.361+0000 7fe6c2ffd640 1 -- 172.21.15.22:0/2651617866 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6bc001c30 con 0x7fe6c4149dc0 2024-03-29T11:37:44.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.361+0000 7fe6c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:44.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.361+0000 7fe6caa55640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:44.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.361+0000 7fe6caa55640 4 set_mon_vals no callback set 2024-03-29T11:37:44.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.362+0000 7fe6caa55640 10 monclient: shutdown 2024-03-29T11:37:44.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.362+0000 7fe6caa55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:44.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.362+0000 7fe6caa55640 1 -- 172.21.15.22:0/2651617866 >> v1:172.21.15.204:6789/0 conn(0x7fe6c4149dc0 legacy=0x7fe6c414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.362+0000 7fe6caa55640 1 -- 172.21.15.22:0/2651617866 shutdown_connections 2024-03-29T11:37:44.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.362+0000 7fe6caa55640 1 -- 172.21.15.22:0/2651617866 >> 172.21.15.22:0/2651617866 conn(0x7fe6c40b35e0 msgr2=0x7fe6c40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:44.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.362+0000 7fe6caa55640 1 -- 172.21.15.22:0/2651617866 shutdown_connections 2024-03-29T11:37:44.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.362+0000 7fe6caa55640 1 -- 172.21.15.22:0/2651617866 wait complete. 2024-03-29T11:37:44.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.363+0000 7fe6caa55640 10 monclient: build_initial_monmap 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.363+0000 7fe6caa55640 10 monclient: monmap: 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.364+0000 7fe6caa55640 1 librados: starting msgr at 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.364+0000 7fe6caa55640 1 librados: starting objecter 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.364+0000 7fe6caa55640 1 Processor -- start 2024-03-29T11:37:44.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.365+0000 7fe6caa55640 1 -- start start 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.365+0000 7fe6caa55640 1 librados: setting wanted keys 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.365+0000 7fe6caa55640 1 librados: calling monclient init 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.365+0000 7fe6caa55640 10 monclient: init 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c40a9930 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c4149dc0 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c414ff40 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c413c0a0 con 0x7fe6c40a9930 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c41e49c0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c413ce40 con 0x7fe6c414ff40 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6caa55640 10 monclient(hunting): authenticate will time out at 1114.643393s 2024-03-29T11:37:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6c8ff2640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe6c414ff40 0x7fe6c41efbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41828/0 (socket says 172.21.15.22:41828) 2024-03-29T11:37:44.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.366+0000 7fe6c8ff2640 1 -- 172.21.15.22:0/3624521155 learned_addr learned my addr 172.21.15.22:0/3624521155 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:44.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.370+0000 7fe6c0ff9640 10 client.?.objecter ms_handle_connect 0x7fe6c414ff40 2024-03-29T11:37:44.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.370+0000 7fe6c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:44.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.370+0000 7fe6c0ff9640 10 client.?.objecter ms_handle_connect 0x7fe6c40a9930 2024-03-29T11:37:44.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.370+0000 7fe6c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:44.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 client.?.objecter ms_handle_connect 0x7fe6c4149dc0 2024-03-29T11:37:44.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:44.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125126059 0 0) 0x7fe6c41e49c0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b0004eb0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165462043 0 0) 0x7fe6c413c0a0 con 0x7fe6c40a9930 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.371+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c41e49c0 con 0x7fe6c40a9930 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853238015 0 0) 0x7fe6c413ce40 con 0x7fe6c414ff40 2024-03-29T11:37:44.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c413c0a0 con 0x7fe6c414ff40 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952069185 0 0) 0x7fe6b0004eb0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6c413ce40 con 0x7fe6c4149dc0 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe6bc001360 con 0x7fe6c4149dc0 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:44.375 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1877671343 0 0) 0x7fe6c413ce40 con 0x7fe6c4149dc0 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 >> v1:172.21.15.22:6790/0 conn(0x7fe6c414ff40 legacy=0x7fe6c41efbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.372+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 >> v1:172.21.15.22:6789/0 conn(0x7fe6c40a9930 legacy=0x7fe6c413c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6c0ff9640 1 monclient: found mon.b 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6c0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c413db00 con 0x7fe6c4149dc0 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6c0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 10 monclient: _renew_subs 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c413ddb0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 10 monclient: _renew_subs 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:44.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c40ac1e0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.373+0000 7fe6caa55640 1 librados: init done 2024-03-29T11:37:44.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.374+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6bc004f70 con 0x7fe6c4149dc0 2024-03-29T11:37:44.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.374+0000 7fe6c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:44.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.374+0000 7fe6c1ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.375+0000 7fe6aa7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.375+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe6bc0053c0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.375+0000 7fe6c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.375+0000 7fe6c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.375+0000 7fe6c0ff9640 10 monclient: dump: 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:44.379 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.375+0000 7fe6aa7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.375+0000 7fe6aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.375+0000 7fe6aa7fc640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c4075f10 con 0x7fe6c4149dc0 2024-03-29T11:37:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.378+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fe6bc001e20 con 0x7fe6c4149dc0 2024-03-29T11:37:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.379+0000 7fe6c3fff640 10 monclient: get_auth_request con 0x7fe6b006dea0 auth_method 0 2024-03-29T11:37:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.379+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fe6bc084da0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.379+0000 7fe6c0ff9640 10 client.4336.objecter ms_dispatch 0x7fe6c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.379+0000 7fe6c0ff9640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:44.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.379+0000 7fe6c0ff9640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:44.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.379+0000 7fe6c0ff9640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T11:37:44.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.379+0000 7fe6c0ff9640 10 client.4336.objecter ms_handle_connect 0x7fe6b006dea0 2024-03-29T11:37:44.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.383+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6bc050680 con 0x7fe6c4149dc0 2024-03-29T11:37:44.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.383+0000 7fe6c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.383+0000 7fe6c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:44.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.731+0000 7fe6aa7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:44.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.731+0000 7fe6aa7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:44.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.731+0000 7fe6aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:44.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.731+0000 7fe6aa7fc640 1 -- 172.21.15.22:0/3624521155 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6c413b120 con 0x7fe6c4149dc0 2024-03-29T11:37:44.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.732+0000 7fe6c0ff9640 1 -- 172.21.15.22:0/3624521155 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fe6bc0525b0 con 0x7fe6c4149dc0 2024-03-29T11:37:44.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.732+0000 7fe6c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:44.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.732+0000 7fe6c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:44.734 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:44.734 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.737+0000 7fe6aa7fc640 10 librados: watch_flush enter 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.737+0000 7fe6aa7fc640 10 librados: watch_flush exit 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.737+0000 7fe6aa7fc640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.737+0000 7fe6aa7fc640 10 client.4336.objecter successfully canceled tick 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.737+0000 7fe6aa7fc640 1 -- 172.21.15.22:0/3624521155 >> v1:172.21.15.204:6816/3848099522 conn(0x7fe6b006dea0 legacy=0x7fe6b0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.737+0000 7fe6aa7fc640 10 monclient: shutdown 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.737+0000 7fe6aa7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.737+0000 7fe6aa7fc640 1 -- 172.21.15.22:0/3624521155 >> v1:172.21.15.204:6789/0 conn(0x7fe6c4149dc0 legacy=0x7fe6c413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.738+0000 7fe6aa7fc640 1 -- 172.21.15.22:0/3624521155 shutdown_connections 2024-03-29T11:37:44.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.738+0000 7fe6aa7fc640 1 -- 172.21.15.22:0/3624521155 >> 172.21.15.22:0/3624521155 conn(0x7fe6c40b35e0 msgr2=0x7fe6c4140a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:44.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.739+0000 7fe6aa7fc640 1 -- 172.21.15.22:0/3624521155 shutdown_connections 2024-03-29T11:37:44.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.739+0000 7fe6aa7fc640 1 -- 172.21.15.22:0/3624521155 wait complete. 2024-03-29T11:37:44.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:44.739+0000 7fe6aa7fc640 1 librados: shutdown 2024-03-29T11:37:44.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:51.564 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.699+0000 7fbad48e9640 10 monclient: get_monmap_and_config 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.699+0000 7fbad48e9640 10 monclient: build_initial_monmap 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.699+0000 7fbad48e9640 10 monclient: monmap: 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:51.701168+0000 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:51.701168+0000 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:51.700 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:51.701 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:51.701 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:51.701 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:51.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.700+0000 7fbad48e9640 1 Processor -- start 2024-03-29T11:37:51.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.700+0000 7fbad48e9640 1 -- start start 2024-03-29T11:37:51.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.700+0000 7fbad48e9640 10 monclient: init 2024-03-29T11:37:51.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.701+0000 7fbad48e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:51.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbad48e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:51.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbad48e9640 10 monclient(hunting): picked mon.noname-a con 0x7fbac8038c60 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:51.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbad48e9640 10 monclient(hunting): picked mon.noname-b con 0x7fbac80324b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:51.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbacf7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fbac8038c60 0x7fbac80c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:54068/0 (socket says 172.21.15.22:54068) 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbaceffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fbac80324b0 0x7fbac80cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:33288/0 (socket says 172.21.15.22:33288) 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbaceffd640 1 -- 172.21.15.22:0/418433119 learned_addr learned my addr 172.21.15.22:0/418433119 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbad48e9640 10 monclient(hunting): picked mon.noname-c con 0x7fbac80d11e0 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbad48e9640 1 -- 172.21.15.22:0/418433119 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac80caf80 con 0x7fbac8038c60 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.702+0000 7fbad48e9640 1 -- 172.21.15.22:0/418433119 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac8039850 con 0x7fbac80d11e0 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbad48e9640 1 -- 172.21.15.22:0/418433119 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac80c78d0 con 0x7fbac80324b0 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbad48e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbad48e9640 10 monclient(hunting): authenticate will time out at 1121.980076s 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111118489 0 0) 0x7fbac80caf80 con 0x7fbac8038c60 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:51.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac0003520 con 0x7fbac8038c60 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018099780 0 0) 0x7fbac80c78d0 con 0x7fbac80324b0 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac00048a0 con 0x7fbac80324b0 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310076072 0 0) 0x7fbac0003520 con 0x7fbac8038c60 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 >> v1:172.21.15.22:6790/0 conn(0x7fbac80324b0 legacy=0x7fbac80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:51.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.703+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 >> v1:172.21.15.204:6789/0 conn(0x7fbac80d11e0 legacy=0x7fbac80d35d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 1 monclient: found mon.noname-a 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac80c7a60 con 0x7fbac8038c60 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbad48e9640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbad48e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:51.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fbac4001670 con 0x7fbac8038c60 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.704+0000 7fbace7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbace7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbace7fc640 10 monclient: dump: 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbad48e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac4001850 con 0x7fbac8038c60 2024-03-29T11:37:51.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbace7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbad48e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbad48e9640 4 set_mon_vals no callback set 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbace7fc640 1 -- 172.21.15.22:0/418433119 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fbac4001c60 con 0x7fbac8038c60 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbace7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbace7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.705+0000 7fbace7fc640 10 monclient: dump: 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:51.709 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:51.710 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:51.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.706+0000 7fbad48e9640 10 monclient: shutdown 2024-03-29T11:37:51.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.706+0000 7fbad48e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:51.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.706+0000 7fbad48e9640 1 -- 172.21.15.22:0/418433119 >> v1:172.21.15.22:6789/0 conn(0x7fbac8038c60 legacy=0x7fbac80c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:51.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.706+0000 7fbad48e9640 1 -- 172.21.15.22:0/418433119 shutdown_connections 2024-03-29T11:37:51.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.706+0000 7fbad48e9640 1 -- 172.21.15.22:0/418433119 >> 172.21.15.22:0/418433119 conn(0x7fbac8005c20 msgr2=0x7fbac8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:51.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.708+0000 7fbad48e9640 1 -- 172.21.15.22:0/418433119 shutdown_connections 2024-03-29T11:37:51.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.708+0000 7fbad48e9640 1 -- 172.21.15.22:0/418433119 wait complete. 2024-03-29T11:37:51.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.710+0000 7fbad48e9640 10 monclient: build_initial_monmap 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.710+0000 7fbad48e9640 10 monclient: monmap: 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.710+0000 7fbad48e9640 1 librados: starting msgr at 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.710+0000 7fbad48e9640 1 librados: starting objecter 2024-03-29T11:37:51.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.711+0000 7fbad48e9640 1 Processor -- start 2024-03-29T11:37:51.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.711+0000 7fbad48e9640 1 -- start start 2024-03-29T11:37:51.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.711+0000 7fbad48e9640 1 librados: setting wanted keys 2024-03-29T11:37:51.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.711+0000 7fbad48e9640 1 librados: calling monclient init 2024-03-29T11:37:51.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.711+0000 7fbad48e9640 10 monclient: init 2024-03-29T11:37:51.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.712+0000 7fbad48e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:51.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.712+0000 7fbad48e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:51.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.712+0000 7fbad48e9640 10 monclient(hunting): picked mon.noname-c con 0x7fbac80324b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:51.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.712+0000 7fbad48e9640 10 monclient(hunting): picked mon.noname-a con 0x7fbac8038c60 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:51.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbaceffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fbac8038c60 0x7fbac815c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:54284/0 (socket says 172.21.15.22:54284) 2024-03-29T11:37:51.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbad48e9640 10 monclient(hunting): picked mon.noname-b con 0x7fbac80d11e0 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:51.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbaceffd640 1 -- 172.21.15.22:0/3075737136 learned_addr learned my addr 172.21.15.22:0/3075737136 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac80358c0 con 0x7fbac8038c60 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac80367c0 con 0x7fbac80d11e0 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac8035400 con 0x7fbac80324b0 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbad48e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbad48e9640 10 monclient(hunting): authenticate will time out at 1121.990473s 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbabbfff640 10 client.?.objecter ms_handle_connect 0x7fbac8038c60 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.713+0000 7fbabbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460361986 0 0) 0x7fbac80358c0 con 0x7fbac8038c60 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:37:51.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab4004eb0 con 0x7fbac8038c60 2024-03-29T11:37:51.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2636384362 0 0) 0x7fbab4004eb0 con 0x7fbac8038c60 2024-03-29T11:37:51.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbac80358c0 con 0x7fbac8038c60 2024-03-29T11:37:51.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fbabc001ad0 con 0x7fbac8038c60 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.714+0000 7fbabbfff640 10 monclient(hunting): dump: 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:51.718 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.715+0000 7fbabbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.715+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 323482504 0 0) 0x7fbac80358c0 con 0x7fbac8038c60 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.715+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 >> v1:172.21.15.22:6790/0 conn(0x7fbac80324b0 legacy=0x7fbac8034cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.715+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 >> v1:172.21.15.204:6789/0 conn(0x7fbac80d11e0 legacy=0x7fbac815d0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 1 monclient: found mon.a 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac8035590 con 0x7fbac8038c60 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 10 monclient: _finish_auth 0 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbabc001d30 con 0x7fbac8038c60 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:51.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fbabc004c30 con 0x7fbac8038c60 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.716+0000 7fbabbfff640 10 monclient: dump: 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:37:51.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 10 monclient: _renew_subs 2024-03-29T11:37:51.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:51.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbacd7fa640 4 set_mon_vals no callback set 2024-03-29T11:37:51.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac80345a0 con 0x7fbac8038c60 2024-03-29T11:37:51.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:51.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 10 monclient: _renew_subs 2024-03-29T11:37:51.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:51.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac81683c0 con 0x7fbac8038c60 2024-03-29T11:37:51.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.717+0000 7fbad48e9640 1 librados: init done 2024-03-29T11:37:51.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.718+0000 7fbad48e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:51.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.719+0000 7fbad48e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:51.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.719+0000 7fbad48e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:51.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.719+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fbabc01ac00 con 0x7fbac8038c60 2024-03-29T11:37:51.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.719+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac8007b90 con 0x7fbac8038c60 2024-03-29T11:37:51.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.720+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fbabc084c30 con 0x7fbac8038c60 2024-03-29T11:37:51.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.720+0000 7fbabbfff640 10 client.4356.objecter ms_dispatch 0x7fbad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:51.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.720+0000 7fbabbfff640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:51.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.720+0000 7fbabbfff640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:51.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.720+0000 7fbacf7fe640 10 monclient: get_auth_request con 0x7fbab406db80 auth_method 0 2024-03-29T11:37:51.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.720+0000 7fbabbfff640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T11:37:51.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.721+0000 7fbabbfff640 10 client.4356.objecter ms_handle_connect 0x7fbab406db80 2024-03-29T11:37:51.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.731+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbabc050510 con 0x7fbac8038c60 2024-03-29T11:37:51.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.731+0000 7fbabbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:51.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:51.731+0000 7fbabbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:52.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.089+0000 7fbad48e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.089+0000 7fbad48e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.089+0000 7fbad48e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:37:52.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.089+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbac8034730 con 0x7fbac8038c60 2024-03-29T11:37:52.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.090+0000 7fbabbfff640 1 -- 172.21.15.22:0/3075737136 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fbabc052440 con 0x7fbac8038c60 2024-03-29T11:37:52.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.090+0000 7fbabbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.090+0000 7fbabbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:52.091 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:52.091 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:52.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.094+0000 7fbad48e9640 10 librados: watch_flush enter 2024-03-29T11:37:52.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.094+0000 7fbad48e9640 10 librados: watch_flush exit 2024-03-29T11:37:52.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.094+0000 7fbad48e9640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T11:37:52.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.094+0000 7fbad48e9640 10 client.4356.objecter successfully canceled tick 2024-03-29T11:37:52.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.096+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 >> v1:172.21.15.204:6816/3848099522 conn(0x7fbab406db80 legacy=0x7fbab4070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.096+0000 7fbad48e9640 10 monclient: shutdown 2024-03-29T11:37:52.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.096+0000 7fbad48e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:52.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.096+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 >> v1:172.21.15.22:6789/0 conn(0x7fbac8038c60 legacy=0x7fbac815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.096+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 shutdown_connections 2024-03-29T11:37:52.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.096+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 >> 172.21.15.22:0/3075737136 conn(0x7fbac8005c20 msgr2=0x7fbac80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:52.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.097+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 shutdown_connections 2024-03-29T11:37:52.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.100+0000 7fbad48e9640 1 -- 172.21.15.22:0/3075737136 wait complete. 2024-03-29T11:37:52.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:52.100+0000 7fbad48e9640 1 librados: shutdown 2024-03-29T11:37:52.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:58.923 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:59.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.068+0000 7f232a50e640 10 monclient: get_monmap_and_config 2024-03-29T11:37:59.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.068+0000 7f232a50e640 10 monclient: build_initial_monmap 2024-03-29T11:37:59.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.068+0000 7f232a50e640 10 monclient: monmap: 2024-03-29T11:37:59.068 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:37:59.069425+0000 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:37:59.069425+0000 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:59.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.069+0000 7f232a50e640 1 Processor -- start 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.069+0000 7f232a50e640 1 -- start start 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.069+0000 7f232a50e640 10 monclient: init 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 10 monclient(hunting): picked mon.noname-b con 0x7f232406c250 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 10 monclient(hunting): picked mon.noname-c con 0x7f2324072400 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 10 monclient(hunting): picked mon.noname-a con 0x7f232406c630 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232406d760 con 0x7f232406c630 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2324070500 con 0x7f2324072400 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23240aff50 con 0x7f232406c250 2024-03-29T11:37:59.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f232a50e640 10 monclient(hunting): authenticate will time out at 1129.347723s 2024-03-29T11:37:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f2328aab640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f232406c630 0x7f2324151ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:49768/0 (socket says 172.21.15.22:49768) 2024-03-29T11:37:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.070+0000 7f2328aab640 1 -- 172.21.15.22:0/2132013164 learned_addr learned my addr 172.21.15.22:0/2132013164 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.071+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151278416 0 0) 0x7f232406d760 con 0x7f232406c630 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.071+0000 7f2322ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.071+0000 7f2322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.071+0000 7f2322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.071+0000 7f2322ffd640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.071+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2310003520 con 0x7f232406c630 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103439285 0 0) 0x7f23240aff50 con 0x7f232406c250 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f232406d760 con 0x7f232406c250 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3376158336 0 0) 0x7f2310003520 con 0x7f232406c630 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 >> v1:172.21.15.22:6790/0 conn(0x7f232406c250 legacy=0x7f2324071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 >> v1:172.21.15.204:6789/0 conn(0x7f2324072400 legacy=0x7f23241465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 monclient: found mon.noname-a 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240b00e0 con 0x7f232406c630 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:59.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f230c001690 con 0x7f232406c630 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f232a50e640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: dump: 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:59.075 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f232a50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f230c0017f0 con 0x7f232406c630 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 1 -- 172.21.15.22:0/2132013164 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f230c001c00 con 0x7f232406c630 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f2322ffd640 10 monclient: dump: 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:59.076 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:59.077 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:59.077 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:59.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f232a50e640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:59.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.072+0000 7f232a50e640 4 set_mon_vals no callback set 2024-03-29T11:37:59.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.073+0000 7f232a50e640 10 monclient: shutdown 2024-03-29T11:37:59.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.073+0000 7f232a50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:59.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.073+0000 7f232a50e640 1 -- 172.21.15.22:0/2132013164 >> v1:172.21.15.22:6789/0 conn(0x7f232406c630 legacy=0x7f2324151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.073+0000 7f232a50e640 1 -- 172.21.15.22:0/2132013164 shutdown_connections 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.073+0000 7f232a50e640 1 -- 172.21.15.22:0/2132013164 >> 172.21.15.22:0/2132013164 conn(0x7f2324067610 msgr2=0x7f23240b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.073+0000 7f232a50e640 1 -- 172.21.15.22:0/2132013164 shutdown_connections 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.073+0000 7f232a50e640 1 -- 172.21.15.22:0/2132013164 wait complete. 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.074+0000 7f232a50e640 10 monclient: build_initial_monmap 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.074+0000 7f232a50e640 10 monclient: monmap: 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:59.082 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.075+0000 7f232a50e640 1 librados: starting msgr at 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.075+0000 7f232a50e640 1 librados: starting objecter 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.075+0000 7f232a50e640 1 Processor -- start 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.075+0000 7f232a50e640 1 -- start start 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.075+0000 7f232a50e640 1 librados: setting wanted keys 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.075+0000 7f232a50e640 1 librados: calling monclient init 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.075+0000 7f232a50e640 10 monclient: init 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 10 monclient(hunting): picked mon.noname-a con 0x7f232406c250 addr v1:172.21.15.22:6789/0 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 10 monclient(hunting): picked mon.noname-b con 0x7f232406c630 addr v1:172.21.15.204:6789/0 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 10 monclient(hunting): picked mon.noname-c con 0x7f2324072400 addr v1:172.21.15.22:6790/0 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23240aa560 con 0x7f232406c250 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f23237fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f232406c630 0x7f23241df3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:57466/0 (socket says 172.21.15.22:57466) 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23241dec40 con 0x7f232406c630 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f23237fe640 1 -- 172.21.15.22:0/2474371458 learned_addr learned my addr 172.21.15.22:0/2474371458 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23241dfac0 con 0x7f2324072400 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f232a50e640 10 monclient(hunting): authenticate will time out at 1129.353695s 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f2320ff9640 10 client.?.objecter ms_handle_connect 0x7f232406c630 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.076+0000 7f2320ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 client.?.objecter ms_handle_connect 0x7f2324072400 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 client.?.objecter ms_handle_connect 0x7f232406c250 2024-03-29T11:37:59.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272315513 0 0) 0x7f23241dec40 con 0x7f232406c630 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304004eb0 con 0x7f232406c630 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812176501 0 0) 0x7f23241dfac0 con 0x7f2324072400 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23241dec40 con 0x7f2324072400 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611605876 0 0) 0x7f23240aa560 con 0x7f232406c250 2024-03-29T11:37:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.077+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23241dfac0 con 0x7f232406c250 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 898155739 0 0) 0x7f2304004eb0 con 0x7f232406c630 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23240aa560 con 0x7f232406c630 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4190732403 0 0) 0x7f23241dec40 con 0x7f2324072400 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2304004eb0 con 0x7f2324072400 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894233571 0 0) 0x7f23241dfac0 con 0x7f232406c250 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23241dec40 con 0x7f232406c250 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f23140017d0 con 0x7f232406c630 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:59.086 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f230c001b00 con 0x7f2324072400 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:59.087 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f231c001c30 con 0x7f232406c250 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:59.088 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3618939610 0 0) 0x7f23240aa560 con 0x7f232406c630 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 >> v1:172.21.15.22:6790/0 conn(0x7f2324072400 legacy=0x7f23240aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 >> v1:172.21.15.22:6789/0 conn(0x7f232406c250 legacy=0x7f23240a9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 1 monclient: found mon.b 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:59.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.078+0000 7f2320ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23241dee90 con 0x7f232406c630 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 10 monclient: _renew_subs 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23240af150 con 0x7f232406c630 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 10 monclient: _renew_subs 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23240b3610 con 0x7f232406c630 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f232a50e640 1 librados: init done 2024-03-29T11:37:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2314001930 con 0x7f232406c630 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2321ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2314004a00 con 0x7f232406c630 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.079+0000 7f2320ff9640 10 monclient: dump: 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:37:59.091 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:37:59.092 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:37:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.080+0000 7f232a50e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.080+0000 7f232a50e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.080+0000 7f232a50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.080+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2308005080 con 0x7f232406c630 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.082+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f2314004b60 con 0x7f232406c630 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.083+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f2314084950 con 0x7f232406c630 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.083+0000 7f2320ff9640 10 client.4339.objecter ms_dispatch 0x7f2324000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.083+0000 7f2320ff9640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.083+0000 7f2320ff9640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.083+0000 7f2320ff9640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.083+0000 7f2323fff640 10 monclient: get_auth_request con 0x7f230406e240 auth_method 0 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.083+0000 7f2320ff9640 10 client.4339.objecter ms_handle_connect 0x7f230406e240 2024-03-29T11:37:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.089+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2314050230 con 0x7f232406c630 2024-03-29T11:37:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.090+0000 7f2320ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.090+0000 7f2320ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:59.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.462+0000 7f232a50e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.462+0000 7f232a50e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.462+0000 7f232a50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:37:59.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.462+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2308005330 con 0x7f232406c630 2024-03-29T11:37:59.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.463+0000 7f2320ff9640 1 -- 172.21.15.22:0/2474371458 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f2314052160 con 0x7f232406c630 2024-03-29T11:37:59.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.463+0000 7f2320ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.463+0000 7f2320ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:59.464 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:37:59.464 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 10 librados: watch_flush enter 2024-03-29T11:37:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 10 librados: watch_flush exit 2024-03-29T11:37:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T11:37:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 10 client.4339.objecter successfully canceled tick 2024-03-29T11:37:59.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 >> v1:172.21.15.204:6816/3848099522 conn(0x7f230406e240 legacy=0x7f2304070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 10 monclient: shutdown 2024-03-29T11:37:59.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:59.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 >> v1:172.21.15.204:6789/0 conn(0x7f232406c630 legacy=0x7f23241df3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 shutdown_connections 2024-03-29T11:37:59.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.467+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 >> 172.21.15.22:0/2474371458 conn(0x7f2324067610 msgr2=0x7f2324149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.468+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 shutdown_connections 2024-03-29T11:37:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.468+0000 7f232a50e640 1 -- 172.21.15.22:0/2474371458 wait complete. 2024-03-29T11:37:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:37:59.468+0000 7f232a50e640 1 librados: shutdown 2024-03-29T11:37:59.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:06.289 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.437+0000 7f7021b65640 10 monclient: get_monmap_and_config 2024-03-29T11:38:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.437+0000 7f7021b65640 10 monclient: build_initial_monmap 2024-03-29T11:38:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.437+0000 7f7021b65640 10 monclient: monmap: 2024-03-29T11:38:06.438 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:06.438 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:06.438 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:38:06.438902+0000 2024-03-29T11:38:06.438 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:38:06.438902+0000 2024-03-29T11:38:06.438 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:06.439 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:06.439 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:06.439 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:38:06.439 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:38:06.439 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.438+0000 7f7021b65640 1 Processor -- start 2024-03-29T11:38:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.438+0000 7f7021b65640 1 -- start start 2024-03-29T11:38:06.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.438+0000 7f7021b65640 10 monclient: init 2024-03-29T11:38:06.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.439+0000 7f7021b65640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.439+0000 7f7021b65640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.439+0000 7f7021b65640 10 monclient(hunting): picked mon.noname-c con 0x7f701c1457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.439+0000 7f7021b65640 10 monclient(hunting): picked mon.noname-b con 0x7f701c13f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.440+0000 7f701affd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f701c13f650 0x7f701c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47178/0 (socket says 172.21.15.22:47178) 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.440+0000 7f7021b65640 10 monclient(hunting): picked mon.noname-a con 0x7f701c0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.440+0000 7f7021b65640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0aaef0 con 0x7f701c0a9d70 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.440+0000 7f701affd640 1 -- 172.21.15.22:0/971224890 learned_addr learned my addr 172.21.15.22:0/971224890 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.440+0000 7f7021b65640 1 -- 172.21.15.22:0/971224890 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c145bd0 con 0x7f701c1457f0 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.440+0000 7f7021b65640 1 -- 172.21.15.22:0/971224890 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c145eb0 con 0x7f701c13f650 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.440+0000 7f7021b65640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.440+0000 7f7021b65640 10 monclient(hunting): authenticate will time out at 1136.717813s 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754152635 0 0) 0x7f701c145eb0 con 0x7f701c13f650 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004003520 con 0x7f701c13f650 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3193180166 0 0) 0x7f7004003520 con 0x7f701c13f650 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 >> v1:172.21.15.204:6789/0 conn(0x7f701c1457f0 legacy=0x7f701c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 >> v1:172.21.15.22:6789/0 conn(0x7f701c0a9d70 legacy=0x7f701c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:38:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c146340 con 0x7f701c13f650 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7010001690 con 0x7f701c13f650 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.441+0000 7f701a7fc640 10 monclient: dump: 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.444 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.442+0000 7f7021b65640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.442+0000 7f7021b65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.442+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7010001860 con 0x7f701c13f650 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.442+0000 7f701a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.442+0000 7f7021b65640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.442+0000 7f7021b65640 4 set_mon_vals no callback set 2024-03-29T11:38:06.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f701a7fc640 1 -- 172.21.15.22:0/971224890 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7010001cb0 con 0x7f701c13f650 2024-03-29T11:38:06.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f701a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f701a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f701a7fc640 10 monclient: dump: 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:06.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f7021b65640 10 monclient: shutdown 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f7021b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f7021b65640 1 -- 172.21.15.22:0/971224890 >> v1:172.21.15.22:6790/0 conn(0x7f701c13f650 legacy=0x7f701c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f7021b65640 1 -- 172.21.15.22:0/971224890 shutdown_connections 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f7021b65640 1 -- 172.21.15.22:0/971224890 >> 172.21.15.22:0/971224890 conn(0x7f701c066a00 msgr2=0x7f701c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f7021b65640 1 -- 172.21.15.22:0/971224890 shutdown_connections 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.443+0000 7f7021b65640 1 -- 172.21.15.22:0/971224890 wait complete. 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.444+0000 7f7021b65640 10 monclient: build_initial_monmap 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.444+0000 7f7021b65640 10 monclient: monmap: 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:06.450 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.444+0000 7f7021b65640 1 librados: starting msgr at 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.444+0000 7f7021b65640 1 librados: starting objecter 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.444+0000 7f7021b65640 1 Processor -- start 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.445+0000 7f7021b65640 1 -- start start 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.445+0000 7f7021b65640 1 librados: setting wanted keys 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.445+0000 7f7021b65640 1 librados: calling monclient init 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.445+0000 7f7021b65640 10 monclient: init 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7021b65640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7021b65640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7021b65640 10 monclient(hunting): picked mon.noname-a con 0x7f701c0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7021b65640 10 monclient(hunting): picked mon.noname-b con 0x7f701c13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:06.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7021b65640 10 monclient(hunting): picked mon.noname-c con 0x7f701c1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f701bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f701c1457f0 0x7f701c1371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47576/0 (socket says 172.21.15.22:47576) 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f701bfff640 1 -- 172.21.15.22:0/1332786688 learned_addr learned my addr 172.21.15.22:0/1332786688 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7003fff640 10 client.?.objecter ms_handle_connect 0x7f701c1457f0 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7003fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c137900 con 0x7f701c0a9d70 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.446+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0ac560 con 0x7f701c13f650 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0ac840 con 0x7f701c1457f0 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7021b65640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7021b65640 10 monclient(hunting): authenticate will time out at 1136.724179s 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7003fff640 10 client.?.objecter ms_handle_connect 0x7f701c13f650 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7003fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572516007 0 0) 0x7f701c0ac840 con 0x7f701c1457f0 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7003fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7003fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7003fff640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:38:06.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.447+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7008004eb0 con 0x7f701c1457f0 2024-03-29T11:38:06.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286936804 0 0) 0x7f701c0ac560 con 0x7f701c13f650 2024-03-29T11:38:06.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:38:06.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c0ac840 con 0x7f701c13f650 2024-03-29T11:38:06.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741858157 0 0) 0x7f701c0ac840 con 0x7f701c13f650 2024-03-29T11:38:06.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f701c0ac560 con 0x7f701c13f650 2024-03-29T11:38:06.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7010001b70 con 0x7f701c13f650 2024-03-29T11:38:06.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 10 monclient(hunting): dump: 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.448+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 42968541 0 0) 0x7f7008004eb0 con 0x7f701c1457f0 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f701c0ac840 con 0x7f701c1457f0 2024-03-29T11:38:06.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7014001c60 con 0x7f701c1457f0 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient(hunting): dump: 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:06.458 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1534113383 0 0) 0x7f701c0ac840 con 0x7f701c1457f0 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 >> v1:172.21.15.204:6789/0 conn(0x7f701c13f650 legacy=0x7f701c136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 >> v1:172.21.15.22:6789/0 conn(0x7f701c0a9d70 legacy=0x7f701c140360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 monclient: found mon.c 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c13fe40 con 0x7f701c1457f0 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7014003970 con 0x7f701c1457f0 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7014003d40 con 0x7f701c1457f0 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7003fff640 10 monclient: dump: 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.449+0000 7f7019ffb640 4 set_mon_vals no callback set 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 10 monclient: _renew_subs 2024-03-29T11:38:06.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c137ba0 con 0x7f701c1457f0 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 10 monclient: _renew_subs 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c0ab620 con 0x7f701c1457f0 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.451+0000 7f7021b65640 1 librados: init done 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.453+0000 7f7021b65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.453+0000 7f7021b65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.453+0000 7f7021b65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.453+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c075f10 con 0x7f701c1457f0 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.453+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f7014003ea0 con 0x7f701c1457f0 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.454+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f7014084da0 con 0x7f701c1457f0 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.454+0000 7f7003fff640 10 client.4358.objecter ms_dispatch 0x7f701c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:06.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.454+0000 7f7003fff640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:06.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.454+0000 7f7003fff640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:06.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.454+0000 7f7003fff640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T11:38:06.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.454+0000 7f701b7fe640 10 monclient: get_auth_request con 0x7f700806de80 auth_method 0 2024-03-29T11:38:06.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.455+0000 7f7003fff640 10 client.4358.objecter ms_handle_connect 0x7f700806de80 2024-03-29T11:38:06.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.460+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7014050680 con 0x7f701c1457f0 2024-03-29T11:38:06.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.460+0000 7f7003fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.460+0000 7f7003fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:06.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.771+0000 7f7021b65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:06.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.771+0000 7f7021b65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:06.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.771+0000 7f7021b65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:06.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.771+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f701c0b3680 con 0x7f701c1457f0 2024-03-29T11:38:06.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.772+0000 7f7003fff640 1 -- 172.21.15.22:0/1332786688 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f70140525b0 con 0x7f701c1457f0 2024-03-29T11:38:06.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.772+0000 7f7003fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:06.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.772+0000 7f7003fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:06.773 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:38:06.773 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:06.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 10 librados: watch_flush enter 2024-03-29T11:38:06.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 10 librados: watch_flush exit 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 10 client.4358.objecter successfully canceled tick 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 >> v1:172.21.15.204:6816/3848099522 conn(0x7f700806de80 legacy=0x7f7008070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 10 monclient: shutdown 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 >> v1:172.21.15.22:6790/0 conn(0x7f701c1457f0 legacy=0x7f701c1371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 shutdown_connections 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.775+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 >> 172.21.15.22:0/1332786688 conn(0x7f701c066a00 msgr2=0x7f701c0b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.776+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 shutdown_connections 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.776+0000 7f7021b65640 1 -- 172.21.15.22:0/1332786688 wait complete. 2024-03-29T11:38:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:06.776+0000 7f7021b65640 1 librados: shutdown 2024-03-29T11:38:06.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:13.598 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:13.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.747+0000 7fe4418c9640 10 monclient: get_monmap_and_config 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.747+0000 7fe4418c9640 10 monclient: build_initial_monmap 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.747+0000 7fe4418c9640 10 monclient: monmap: 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:38:13.748438+0000 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:38:13.748438+0000 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:38:13.748 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:13.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.749+0000 7fe4418c9640 1 Processor -- start 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.749+0000 7fe4418c9640 1 -- start start 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.749+0000 7fe4418c9640 10 monclient: init 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe43c0b0770 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe43c14a700 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe43c0ac5a0 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:13.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c0aba70 con 0x7fe43c0ac5a0 2024-03-29T11:38:13.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c0aecb0 con 0x7fe43c0b0770 2024-03-29T11:38:13.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c0af150 con 0x7fe43c14a700 2024-03-29T11:38:13.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:13.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.750+0000 7fe4418c9640 10 monclient(hunting): authenticate will time out at 1144.027532s 2024-03-29T11:38:13.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.751+0000 7fe43a7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe43c14a700 0x7fe43c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41086/0 (socket says 172.21.15.22:41086) 2024-03-29T11:38:13.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.751+0000 7fe43a7fc640 1 -- 172.21.15.22:0/1956269393 learned_addr learned my addr 172.21.15.22:0/1956269393 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.752+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388949497 0 0) 0x7fe43c0af150 con 0x7fe43c14a700 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.752+0000 7fe439ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.752+0000 7fe439ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.752+0000 7fe439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.752+0000 7fe439ffb640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.752+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe428003520 con 0x7fe43c14a700 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.752+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 887545730 0 0) 0x7fe428003520 con 0x7fe43c14a700 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.752+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 >> v1:172.21.15.204:6789/0 conn(0x7fe43c0b0770 legacy=0x7fe43c149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 >> v1:172.21.15.22:6789/0 conn(0x7fe43c0ac5a0 legacy=0x7fe43c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:13.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 1 monclient: found mon.noname-b 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c0ac9d0 con 0x7fe43c14a700 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe424001690 con 0x7fe43c14a700 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe439ffb640 10 monclient: dump: 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:13.757 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe4418c9640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.753+0000 7fe4418c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.754+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4240018f0 con 0x7fe43c14a700 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.754+0000 7fe439ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.754+0000 7fe439ffb640 1 -- 172.21.15.22:0/1956269393 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe424001d40 con 0x7fe43c14a700 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.754+0000 7fe439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.754+0000 7fe439ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:13.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.754+0000 7fe439ffb640 10 monclient: dump: 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.754+0000 7fe4418c9640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.754+0000 7fe4418c9640 4 set_mon_vals no callback set 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.755+0000 7fe4418c9640 10 monclient: shutdown 2024-03-29T11:38:13.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.755+0000 7fe4418c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:13.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.755+0000 7fe4418c9640 1 -- 172.21.15.22:0/1956269393 >> v1:172.21.15.22:6790/0 conn(0x7fe43c14a700 legacy=0x7fe43c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.755+0000 7fe4418c9640 1 -- 172.21.15.22:0/1956269393 shutdown_connections 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.755+0000 7fe4418c9640 1 -- 172.21.15.22:0/1956269393 >> 172.21.15.22:0/1956269393 conn(0x7fe43c06b660 msgr2=0x7fe43c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.756+0000 7fe4418c9640 1 -- 172.21.15.22:0/1956269393 shutdown_connections 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.756+0000 7fe4418c9640 1 -- 172.21.15.22:0/1956269393 wait complete. 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.757+0000 7fe4418c9640 10 monclient: build_initial_monmap 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.757+0000 7fe4418c9640 10 monclient: monmap: 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:38:13.761 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.757+0000 7fe4418c9640 1 librados: starting msgr at 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.757+0000 7fe4418c9640 1 librados: starting objecter 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.757+0000 7fe4418c9640 1 Processor -- start 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.758+0000 7fe4418c9640 1 -- start start 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.758+0000 7fe4418c9640 1 librados: setting wanted keys 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.758+0000 7fe4418c9640 1 librados: calling monclient init 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.758+0000 7fe4418c9640 10 monclient: init 2024-03-29T11:38:13.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.758+0000 7fe4418c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.758+0000 7fe4418c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.759+0000 7fe4418c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe43c0ac5a0 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.759+0000 7fe4418c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe43c0b0770 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.759+0000 7fe4418c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe43c14a700 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.759+0000 7fe4418c9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c1259b0 con 0x7fe43c14a700 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.759+0000 7fe4418c9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c12a8f0 con 0x7fe43c0ac5a0 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.759+0000 7fe4418c9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c128840 con 0x7fe43c0b0770 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.759+0000 7fe4418c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.759+0000 7fe4418c9640 10 monclient(hunting): authenticate will time out at 1144.036096s 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.760+0000 7fe43a7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe43c0b0770 0x7fe43c147200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41500/0 (socket says 172.21.15.22:41500) 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.760+0000 7fe43a7fc640 1 -- 172.21.15.22:0/3717790510 learned_addr learned my addr 172.21.15.22:0/3717790510 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.760+0000 7fe433fff640 10 client.?.objecter ms_handle_connect 0x7fe43c0b0770 2024-03-29T11:38:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.760+0000 7fe433fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.762+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728972199 0 0) 0x7fe43c128840 con 0x7fe43c0b0770 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.762+0000 7fe433fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.762+0000 7fe433fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.762+0000 7fe433fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.762+0000 7fe433fff640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.762+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe41c004eb0 con 0x7fe43c0b0770 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.763+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222256997 0 0) 0x7fe41c004eb0 con 0x7fe43c0b0770 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.763+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe41c005300 con 0x7fe43c0b0770 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.763+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe424001650 con 0x7fe43c0b0770 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.763+0000 7fe433fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.763+0000 7fe433fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.763+0000 7fe433fff640 10 monclient(hunting): dump: 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:13.768 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.763+0000 7fe433fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 10 client.?.objecter ms_handle_connect 0x7fe43c14a700 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 880725283 0 0) 0x7fe41c005300 con 0x7fe43c0b0770 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 >> v1:172.21.15.204:6789/0 conn(0x7fe43c0ac5a0 legacy=0x7fe43c128d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 >> v1:172.21.15.22:6789/0 conn(0x7fe43c14a700 legacy=0x7fe43c1e71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 1 monclient: found mon.c 2024-03-29T11:38:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c1dfab0 con 0x7fe43c0b0770 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe433fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe4418c9640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe4418c9640 10 monclient: _renew_subs 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe4418c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.764+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43c1dfd60 con 0x7fe43c0b0770 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe4418c9640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe4418c9640 10 monclient: _renew_subs 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe4418c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43c1284d0 con 0x7fe43c0b0770 2024-03-29T11:38:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe4418c9640 1 librados: init done 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe424004b10 con 0x7fe43c0b0770 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe433fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe424001650 con 0x7fe43c0b0770 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe433fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe433fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe433fff640 10 monclient: dump: 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:13.771 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.765+0000 7fe438ff9640 4 set_mon_vals no callback set 2024-03-29T11:38:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.768+0000 7fe4418c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.768+0000 7fe4418c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.768+0000 7fe4418c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.768+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe420005080 con 0x7fe43c0b0770 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.769+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fe424004e00 con 0x7fe43c0b0770 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.770+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fe4240845e0 con 0x7fe43c0b0770 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.770+0000 7fe433fff640 10 client.4364.objecter ms_dispatch 0x7fe43c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.770+0000 7fe433fff640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.770+0000 7fe433fff640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.770+0000 7fe433fff640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.770+0000 7fe43affd640 10 monclient: get_auth_request con 0x7fe41c06e2a0 auth_method 0 2024-03-29T11:38:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.771+0000 7fe433fff640 10 client.4364.objecter ms_handle_connect 0x7fe41c06e2a0 2024-03-29T11:38:13.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.776+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe42404fec0 con 0x7fe43c0b0770 2024-03-29T11:38:13.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.776+0000 7fe433fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:13.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:13.776+0000 7fe433fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:14.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.102+0000 7fe4418c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.102+0000 7fe4418c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.102+0000 7fe4418c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:14.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.102+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe420005330 con 0x7fe43c0b0770 2024-03-29T11:38:14.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.104+0000 7fe433fff640 1 -- 172.21.15.22:0/3717790510 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fe42404fec0 con 0x7fe43c0b0770 2024-03-29T11:38:14.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.104+0000 7fe433fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.104+0000 7fe433fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:14.105 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:38:14.106 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:14.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.107+0000 7fe4418c9640 10 librados: watch_flush enter 2024-03-29T11:38:14.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 10 librados: watch_flush exit 2024-03-29T11:38:14.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:38:14.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 >> v1:172.21.15.204:6816/3848099522 conn(0x7fe41c06e2a0 legacy=0x7fe41c070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 10 monclient: shutdown 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 >> v1:172.21.15.22:6790/0 conn(0x7fe43c0b0770 legacy=0x7fe43c147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 shutdown_connections 2024-03-29T11:38:14.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.108+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 >> 172.21.15.22:0/3717790510 conn(0x7fe43c06b660 msgr2=0x7fe43c0b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:14.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.109+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 shutdown_connections 2024-03-29T11:38:14.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.109+0000 7fe4418c9640 1 -- 172.21.15.22:0/3717790510 wait complete. 2024-03-29T11:38:14.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:14.110+0000 7fe4418c9640 1 librados: shutdown 2024-03-29T11:38:14.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:20.936 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.086+0000 7fa383cc3640 10 monclient: get_monmap_and_config 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.086+0000 7fa383cc3640 10 monclient: build_initial_monmap 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.087+0000 7fa383cc3640 10 monclient: monmap: 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:38:21.087264+0000 2024-03-29T11:38:21.088 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:38:21.087264+0000 2024-03-29T11:38:21.089 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:21.089 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:21.089 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:21.089 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:38:21.089 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:38:21.089 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:21.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.088+0000 7fa383cc3640 1 Processor -- start 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.088+0000 7fa383cc3640 1 -- start start 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.088+0000 7fa383cc3640 10 monclient: init 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 10 monclient(hunting): picked mon.noname-c con 0x7fa37c0a9ac0 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 10 monclient(hunting): picked mon.noname-b con 0x7fa37c13ee90 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 10 monclient(hunting): picked mon.noname-a con 0x7fa37c0af900 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:21.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0afd10 con 0x7fa37c0af900 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0af6f0 con 0x7fa37c0a9ac0 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0aac80 con 0x7fa37c13ee90 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.089+0000 7fa383cc3640 10 monclient(hunting): authenticate will time out at 1151.365585s 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.090+0000 7fa38125e640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa37c13ee90 0x7fa37c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57302/0 (socket says 172.21.15.22:57302) 2024-03-29T11:38:21.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.090+0000 7fa38125e640 1 -- 172.21.15.22:0/2583386340 learned_addr learned my addr 172.21.15.22:0/2583386340 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.091+0000 7fa380a5d640 1 -- 172.21.15.22:0/2583386340 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096219508 0 0) 0x7fa37c0aac80 con 0x7fa37c13ee90 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.091+0000 7fa380a5d640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.091+0000 7fa380a5d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.091+0000 7fa380a5d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.091+0000 7fa380a5d640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.091+0000 7fa380a5d640 1 -- 172.21.15.22:0/2583386340 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa368003520 con 0x7fa37c13ee90 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.092+0000 7fa380a5d640 1 -- 172.21.15.22:0/2583386340 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336182094 0 0) 0x7fa368003520 con 0x7fa37c13ee90 2024-03-29T11:38:21.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.092+0000 7fa380a5d640 1 -- 172.21.15.22:0/2583386340 >> v1:172.21.15.204:6789/0 conn(0x7fa37c0a9ac0 legacy=0x7fa37c0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:21.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.092+0000 7fa380a5d640 1 -- 172.21.15.22:0/2583386340 >> v1:172.21.15.22:6789/0 conn(0x7fa37c0af900 legacy=0x7fa37c149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 10 monclient: _finish_hunting 0 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 1 monclient: found mon.noname-b 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 1 -- 172.21.15.22:0/2583386340 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0aaf60 con 0x7fa37c13ee90 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 10 monclient: _finish_auth 0 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 10 monclient: _check_auth_tickets 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa383cc3640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa383cc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 1 -- 172.21.15.22:0/2583386340 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa374001a10 con 0x7fa37c13ee90 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:21.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa380a5d640 10 monclient: dump: 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:21.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.093+0000 7fa383cc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:21.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.096+0000 7fa380a5d640 1 -- 172.21.15.22:0/2583386340 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa374001ca0 con 0x7fa37c13ee90 2024-03-29T11:38:21.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.096+0000 7fa380a5d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:21.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.096+0000 7fa383cc3640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.096+0000 7fa383cc3640 4 set_mon_vals no callback set 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.097+0000 7fa383cc3640 10 monclient: shutdown 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.097+0000 7fa383cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.097+0000 7fa383cc3640 1 -- 172.21.15.22:0/2583386340 >> v1:172.21.15.22:6790/0 conn(0x7fa37c13ee90 legacy=0x7fa37c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.098+0000 7fa383cc3640 1 -- 172.21.15.22:0/2583386340 shutdown_connections 2024-03-29T11:38:21.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.098+0000 7fa383cc3640 1 -- 172.21.15.22:0/2583386340 >> 172.21.15.22:0/2583386340 conn(0x7fa37c0b35e0 msgr2=0x7fa37c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:21.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.100+0000 7fa383cc3640 1 -- 172.21.15.22:0/2583386340 shutdown_connections 2024-03-29T11:38:21.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.101+0000 7fa383cc3640 1 -- 172.21.15.22:0/2583386340 wait complete. 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.102+0000 7fa383cc3640 10 monclient: build_initial_monmap 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.102+0000 7fa383cc3640 10 monclient: monmap: 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:21.102 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.102+0000 7fa383cc3640 1 librados: starting msgr at 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.102+0000 7fa383cc3640 1 librados: starting objecter 2024-03-29T11:38:21.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.102+0000 7fa383cc3640 1 Processor -- start 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 1 -- start start 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 1 librados: setting wanted keys 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 1 librados: calling monclient init 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 10 monclient: init 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 10 monclient(hunting): picked mon.noname-a con 0x7fa37c0a9ac0 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 10 monclient(hunting): picked mon.noname-b con 0x7fa37c0af900 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 10 monclient(hunting): picked mon.noname-c con 0x7fa37c13ee90 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c070e90 con 0x7fa37c0a9ac0 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c1dfc70 con 0x7fa37c0af900 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c06fcc0 con 0x7fa37c13ee90 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.103+0000 7fa383cc3640 10 monclient(hunting): authenticate will time out at 1151.380006s 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.105+0000 7fa382260640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa37c13ee90 0x7fa37c132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58036/0 (socket says 172.21.15.22:58036) 2024-03-29T11:38:21.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.105+0000 7fa382260640 1 -- 172.21.15.22:0/502284480 learned_addr learned my addr 172.21.15.22:0/502284480 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.105+0000 7fa3727fc640 10 client.?.objecter ms_handle_connect 0x7fa37c13ee90 2024-03-29T11:38:21.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.105+0000 7fa3727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.107+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153376933 0 0) 0x7fa37c06fcc0 con 0x7fa37c13ee90 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.107+0000 7fa3727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.107+0000 7fa3727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.107+0000 7fa3727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.107+0000 7fa3727fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:38:21.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.107+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa360004eb0 con 0x7fa37c13ee90 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.107+0000 7fa3727fc640 10 client.?.objecter ms_handle_connect 0x7fa37c0a9ac0 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.107+0000 7fa3727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 client.?.objecter ms_handle_connect 0x7fa37c0af900 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054945978 0 0) 0x7fa37c070e90 con 0x7fa37c0a9ac0 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa37c06fcc0 con 0x7fa37c0a9ac0 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3369699895 0 0) 0x7fa360004eb0 con 0x7fa37c13ee90 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa37c070e90 con 0x7fa37c13ee90 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa36c0018e0 con 0x7fa37c13ee90 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): dump: 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:21.109 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:21.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 128554309 0 0) 0x7fa37c06fcc0 con 0x7fa37c0a9ac0 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa360004eb0 con 0x7fa37c0a9ac0 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa378001770 con 0x7fa37c0a9ac0 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient(hunting): dump: 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.111 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301889060 0 0) 0x7fa37c070e90 con 0x7fa37c13ee90 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 >> v1:172.21.15.204:6789/0 conn(0x7fa37c0af900 legacy=0x7fa37c06f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 >> v1:172.21.15.22:6789/0 conn(0x7fa37c0a9ac0 legacy=0x7fa37c1df560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 monclient: found mon.c 2024-03-29T11:38:21.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c06fe50 con 0x7fa37c13ee90 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa3727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa383cc3640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa383cc3640 10 monclient: _renew_subs 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa383cc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:21.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.108+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37c06ea50 con 0x7fa37c13ee90 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa383cc3640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa383cc3640 10 monclient: _renew_subs 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa383cc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37c071240 con 0x7fa37c13ee90 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa36c0035f0 con 0x7fa37c13ee90 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa383cc3640 1 librados: init done 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa3727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa36c003940 con 0x7fa37c13ee90 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa3727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa3727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa3727fc640 10 monclient: dump: 2024-03-29T11:38:21.116 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.109+0000 7fa3737fe640 4 set_mon_vals no callback set 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.111+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fa36c003aa0 con 0x7fa37c13ee90 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fa36c084950 con 0x7fa37c13ee90 2024-03-29T11:38:21.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa381a5f640 10 monclient: get_auth_request con 0x7fa36006dee0 auth_method 0 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa383cc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa383cc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa383cc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37c075f10 con 0x7fa37c13ee90 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa3727fc640 10 client.4370.objecter ms_dispatch 0x7fa37c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa3727fc640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa3727fc640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.112+0000 7fa3727fc640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:38:21.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.113+0000 7fa3727fc640 10 client.4370.objecter ms_handle_connect 0x7fa36006dee0 2024-03-29T11:38:21.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.121+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa36c050230 con 0x7fa37c13ee90 2024-03-29T11:38:21.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.121+0000 7fa3727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.121+0000 7fa3727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:21.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.457+0000 7fa383cc3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.457+0000 7fa383cc3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.457+0000 7fa383cc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:21.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.457+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa37c1df180 con 0x7fa37c13ee90 2024-03-29T11:38:21.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.458+0000 7fa3727fc640 1 -- 172.21.15.22:0/502284480 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fa36c052160 con 0x7fa37c13ee90 2024-03-29T11:38:21.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.458+0000 7fa3727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.458+0000 7fa3727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:21.458 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:38:21.458 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:21.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 10 librados: watch_flush enter 2024-03-29T11:38:21.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 10 librados: watch_flush exit 2024-03-29T11:38:21.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 >> v1:172.21.15.204:6816/3848099522 conn(0x7fa36006dee0 legacy=0x7fa3600703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 10 monclient: shutdown 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 >> v1:172.21.15.22:6790/0 conn(0x7fa37c13ee90 legacy=0x7fa37c132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 shutdown_connections 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.461+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 >> 172.21.15.22:0/502284480 conn(0x7fa37c0b35e0 msgr2=0x7fa37c0b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.462+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 shutdown_connections 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.462+0000 7fa383cc3640 1 -- 172.21.15.22:0/502284480 wait complete. 2024-03-29T11:38:21.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:21.462+0000 7fa383cc3640 1 librados: shutdown 2024-03-29T11:38:21.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:28.285 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:28.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.435+0000 7f42d367b640 10 monclient: get_monmap_and_config 2024-03-29T11:38:28.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.435+0000 7f42d367b640 10 monclient: build_initial_monmap 2024-03-29T11:38:28.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.435+0000 7f42d367b640 10 monclient: monmap: 2024-03-29T11:38:28.435 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:28.435 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:28.435 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:38:28.435575+0000 2024-03-29T11:38:28.435 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:38:28.435575+0000 2024-03-29T11:38:28.435 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:28.436 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:28.436 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:28.436 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:38:28.436 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:38:28.436 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:28.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.437+0000 7f42d367b640 1 Processor -- start 2024-03-29T11:38:28.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.437+0000 7f42d367b640 1 -- start start 2024-03-29T11:38:28.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.437+0000 7f42d367b640 10 monclient: init 2024-03-29T11:38:28.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 10 monclient(hunting): picked mon.noname-a con 0x7f42cc0ae9b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 10 monclient(hunting): picked mon.noname-b con 0x7f42cc143f50 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 10 monclient(hunting): picked mon.noname-c con 0x7f42cc0b4890 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc0aed90 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc0af020 con 0x7f42cc0b4890 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc0af300 con 0x7f42cc143f50 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d367b640 10 monclient(hunting): authenticate will time out at 1158.714603s 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d0c16640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f42cc143f50 0x7f42cc0b4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34270/0 (socket says 172.21.15.22:34270) 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.438+0000 7f42d0c16640 1 -- 172.21.15.22:0/862047316 learned_addr learned my addr 172.21.15.22:0/862047316 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.439+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182624808 0 0) 0x7f42cc0af300 con 0x7f42cc143f50 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.439+0000 7f42c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.439+0000 7f42c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.439+0000 7f42c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.439+0000 7f42c3fff640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:38:28.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.439+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b8003520 con 0x7f42cc143f50 2024-03-29T11:38:28.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.439+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1363022623 0 0) 0x7f42b8003520 con 0x7f42cc143f50 2024-03-29T11:38:28.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.439+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 >> v1:172.21.15.204:6789/0 conn(0x7f42cc0b4890 legacy=0x7f42cc151df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:28.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 >> v1:172.21.15.22:6789/0 conn(0x7f42cc0ae9b0 legacy=0x7f42cc143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 1 monclient: found mon.noname-b 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc0b0710 con 0x7f42cc143f50 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42d367b640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42d367b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f42bc001690 con 0x7f42cc143f50 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:28.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.443+0000 7f42c3fff640 10 monclient: dump: 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.444+0000 7f42d367b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.444+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42bc0017f0 con 0x7f42cc143f50 2024-03-29T11:38:28.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.444+0000 7f42c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.444+0000 7f42d367b640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.444+0000 7f42d367b640 4 set_mon_vals no callback set 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.444+0000 7f42c3fff640 1 -- 172.21.15.22:0/862047316 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f42bc001c40 con 0x7f42cc143f50 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.444+0000 7f42c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.445+0000 7f42c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.445+0000 7f42c3fff640 10 monclient: dump: 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:28.448 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.445+0000 7f42d367b640 10 monclient: shutdown 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.445+0000 7f42d367b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.445+0000 7f42d367b640 1 -- 172.21.15.22:0/862047316 >> v1:172.21.15.22:6790/0 conn(0x7f42cc143f50 legacy=0x7f42cc0b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.445+0000 7f42d367b640 1 -- 172.21.15.22:0/862047316 shutdown_connections 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.445+0000 7f42d367b640 1 -- 172.21.15.22:0/862047316 >> 172.21.15.22:0/862047316 conn(0x7f42cc066a00 msgr2=0x7f42cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:28.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.446+0000 7f42d367b640 1 -- 172.21.15.22:0/862047316 shutdown_connections 2024-03-29T11:38:28.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.446+0000 7f42d367b640 1 -- 172.21.15.22:0/862047316 wait complete. 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 10 monclient: build_initial_monmap 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 10 monclient: monmap: 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:38:28.452 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 1 librados: starting msgr at 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 1 librados: starting objecter 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 1 Processor -- start 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 1 -- start start 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 1 librados: setting wanted keys 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 1 librados: calling monclient init 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.447+0000 7f42d367b640 10 monclient: init 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 10 monclient(hunting): picked mon.noname-c con 0x7f42cc0ae9b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 10 monclient(hunting): picked mon.noname-b con 0x7f42cc0b4890 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:28.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 10 monclient(hunting): picked mon.noname-a con 0x7f42cc143f50 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:28.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc1db660 con 0x7f42cc143f50 2024-03-29T11:38:28.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc070b00 con 0x7f42cc0b4890 2024-03-29T11:38:28.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc06f8e0 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:28.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.448+0000 7f42d367b640 10 monclient(hunting): authenticate will time out at 1158.724995s 2024-03-29T11:38:28.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.452+0000 7f42d1c18640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f42cc143f50 0x7f42cc1e6f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:54609/0 (socket says 172.21.15.22:54609) 2024-03-29T11:38:28.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.452+0000 7f42d1c18640 1 -- 172.21.15.22:0/3147338184 learned_addr learned my addr 172.21.15.22:0/3147338184 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:28.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.452+0000 7f42c1ffb640 10 client.?.objecter ms_handle_connect 0x7f42cc143f50 2024-03-29T11:38:28.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.452+0000 7f42c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 client.?.objecter ms_handle_connect 0x7f42cc0b4890 2024-03-29T11:38:28.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 client.?.objecter ms_handle_connect 0x7f42cc0ae9b0 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029105794 0 0) 0x7f42cc1db660 con 0x7f42cc143f50 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b0004eb0 con 0x7f42cc143f50 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000450079 0 0) 0x7f42cc06f8e0 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42cc1db660 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489085203 0 0) 0x7f42cc070b00 con 0x7f42cc0b4890 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.453+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42cc06f8e0 con 0x7f42cc0b4890 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3807031666 0 0) 0x7f42cc1db660 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42cc070b00 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f42c4001a40 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:28.461 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 76701603 0 0) 0x7f42cc070b00 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 >> v1:172.21.15.204:6789/0 conn(0x7f42cc0b4890 legacy=0x7f42cc071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 >> v1:172.21.15.22:6789/0 conn(0x7f42cc143f50 legacy=0x7f42cc1e6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 1 monclient: found mon.c 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc1d91a0 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 10 monclient: _renew_subs 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc06e6a0 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 10 monclient: _renew_subs 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:28.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42cc1dc360 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.454+0000 7f42d367b640 1 librados: init done 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.456+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42c4001f30 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.456+0000 7f42c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.456+0000 7f42c2ffd640 4 set_mon_vals no callback set 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.456+0000 7f42d367b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.457+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f42c4003b10 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.457+0000 7f42c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.457+0000 7f42c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.457+0000 7f42c1ffb640 10 monclient: dump: 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:28.464 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.458+0000 7f42d367b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.458+0000 7f42d367b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.458+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ac005080 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.458+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f42c4019ae0 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.459+0000 7f42d0c16640 10 monclient: get_auth_request con 0x7f42b006def0 auth_method 0 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.459+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f42c4083990 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.459+0000 7f42c1ffb640 10 client.4376.objecter ms_dispatch 0x7f42cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.459+0000 7f42c1ffb640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.459+0000 7f42c1ffb640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:28.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.459+0000 7f42c1ffb640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:38:28.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.460+0000 7f42c1ffb640 10 client.4376.objecter ms_handle_connect 0x7f42b006def0 2024-03-29T11:38:28.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.467+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42c404f320 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.467+0000 7f42c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.467+0000 7f42c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:28.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.799+0000 7f42d367b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:28.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.799+0000 7f42d367b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:28.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.799+0000 7f42d367b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:28.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.799+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42ac005330 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.801+0000 7f42c1ffb640 1 -- 172.21.15.22:0/3147338184 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f42c4051250 con 0x7f42cc0ae9b0 2024-03-29T11:38:28.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.801+0000 7f42c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:28.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.801+0000 7f42c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:28.801 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:38:28.801 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 10 librados: watch_flush enter 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 10 librados: watch_flush exit 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 >> v1:172.21.15.204:6816/3848099522 conn(0x7f42b006def0 legacy=0x7f42b00703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 10 monclient: shutdown 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 >> v1:172.21.15.22:6790/0 conn(0x7f42cc0ae9b0 legacy=0x7f42cc1dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 shutdown_connections 2024-03-29T11:38:28.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.805+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 >> 172.21.15.22:0/3147338184 conn(0x7f42cc066a00 msgr2=0x7f42cc0b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:28.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.806+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 shutdown_connections 2024-03-29T11:38:28.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.806+0000 7f42d367b640 1 -- 172.21.15.22:0/3147338184 wait complete. 2024-03-29T11:38:28.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:28.806+0000 7f42d367b640 1 librados: shutdown 2024-03-29T11:38:28.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:35.632 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:35.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.778+0000 7fef0e625640 10 monclient: get_monmap_and_config 2024-03-29T11:38:35.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.778+0000 7fef0e625640 10 monclient: build_initial_monmap 2024-03-29T11:38:35.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.778+0000 7fef0e625640 10 monclient: monmap: 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:38:35.779601+0000 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:38:35.779601+0000 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:38:35.780 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:35.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.780+0000 7fef0e625640 1 Processor -- start 2024-03-29T11:38:35.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.780+0000 7fef0e625640 1 -- start start 2024-03-29T11:38:35.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.780+0000 7fef0e625640 10 monclient: init 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 10 monclient(hunting): picked mon.noname-c con 0x7fef080b0770 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 10 monclient(hunting): picked mon.noname-b con 0x7fef0814a700 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 10 monclient(hunting): picked mon.noname-a con 0x7fef080ac5a0 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080aba70 con 0x7fef080ac5a0 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080aecb0 con 0x7fef080b0770 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080af150 con 0x7fef0814a700 2024-03-29T11:38:35.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:35.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.781+0000 7fef0e625640 10 monclient(hunting): authenticate will time out at 1166.058252s 2024-03-29T11:38:35.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.787+0000 7fef07fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fef080b0770 0x7fef08149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:56651/0 (socket says 172.21.15.22:56651) 2024-03-29T11:38:35.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.787+0000 7fef07fff640 1 -- 172.21.15.22:0/421042643 learned_addr learned my addr 172.21.15.22:0/421042643 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:35.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.787+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276984618 0 0) 0x7fef080aecb0 con 0x7fef080b0770 2024-03-29T11:38:35.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.787+0000 7fef06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:35.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.787+0000 7fef06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:35.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.787+0000 7fef06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:35.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.787+0000 7fef06ffd640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:38:35.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef0003520 con 0x7fef080b0770 2024-03-29T11:38:35.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216712074 0 0) 0x7fef080aba70 con 0x7fef080ac5a0 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef080aecb0 con 0x7fef080ac5a0 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1979775584 0 0) 0x7feef0003520 con 0x7fef080b0770 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 >> v1:172.21.15.22:6790/0 conn(0x7fef0814a700 legacy=0x7fef080abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 >> v1:172.21.15.22:6789/0 conn(0x7fef080ac5a0 legacy=0x7fef08151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 monclient: found mon.noname-c 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080ac9d0 con 0x7fef080b0770 2024-03-29T11:38:35.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.788+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fef00001a30 con 0x7fef080b0770 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.789+0000 7fef06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.789+0000 7fef06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.789+0000 7fef06ffd640 10 monclient: dump: 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:35.791 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:35.792 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:35.792 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:35.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.789+0000 7fef0e625640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:38:35.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.789+0000 7fef0e625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:35.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.794+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef00001b90 con 0x7fef080b0770 2024-03-29T11:38:35.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.794+0000 7fef06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.794+0000 7fef06ffd640 1 -- 172.21.15.22:0/421042643 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fef00001fa0 con 0x7fef080b0770 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.794+0000 7fef06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.794+0000 7fef06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.794+0000 7fef06ffd640 10 monclient: dump: 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:35.796 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:35.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.794+0000 7fef0e625640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:35.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.794+0000 7fef0e625640 4 set_mon_vals no callback set 2024-03-29T11:38:35.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.795+0000 7fef0e625640 10 monclient: shutdown 2024-03-29T11:38:35.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.795+0000 7fef0e625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:35.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.795+0000 7fef0e625640 1 -- 172.21.15.22:0/421042643 >> v1:172.21.15.204:6789/0 conn(0x7fef080b0770 legacy=0x7fef08149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:35.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.797+0000 7fef0e625640 1 -- 172.21.15.22:0/421042643 shutdown_connections 2024-03-29T11:38:35.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.797+0000 7fef0e625640 1 -- 172.21.15.22:0/421042643 >> 172.21.15.22:0/421042643 conn(0x7fef0806b660 msgr2=0x7fef0806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:35.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.798+0000 7fef0e625640 1 -- 172.21.15.22:0/421042643 shutdown_connections 2024-03-29T11:38:35.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.799+0000 7fef0e625640 1 -- 172.21.15.22:0/421042643 wait complete. 2024-03-29T11:38:35.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 10 monclient: build_initial_monmap 2024-03-29T11:38:35.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 10 monclient: monmap: 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 1 librados: starting msgr at 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 1 librados: starting objecter 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 1 Processor -- start 2024-03-29T11:38:35.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 1 -- start start 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 1 librados: setting wanted keys 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 1 librados: calling monclient init 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.800+0000 7fef0e625640 10 monclient: init 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 10 monclient(hunting): picked mon.noname-c con 0x7fef080ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 10 monclient(hunting): picked mon.noname-a con 0x7fef080b0770 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 10 monclient(hunting): picked mon.noname-b con 0x7fef0814a700 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef081eca20 con 0x7fef080b0770 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef081ed5c0 con 0x7fef0814a700 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0807f810 con 0x7fef080ac5a0 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:35.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.801+0000 7fef0e625640 10 monclient(hunting): authenticate will time out at 1166.078818s 2024-03-29T11:38:35.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.805+0000 7fef07fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fef080ac5a0 0x7fef0807e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:48627/0 (socket says 172.21.15.22:48627) 2024-03-29T11:38:35.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.805+0000 7fef07fff640 1 -- 172.21.15.22:0/2759703081 learned_addr learned my addr 172.21.15.22:0/2759703081 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:35.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.807+0000 7fef04ff9640 10 client.?.objecter ms_handle_connect 0x7fef080ac5a0 2024-03-29T11:38:35.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.807+0000 7fef04ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:35.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.807+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939857654 0 0) 0x7fef0807f810 con 0x7fef080ac5a0 2024-03-29T11:38:35.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.807+0000 7fef04ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:35.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.807+0000 7fef04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:35.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.807+0000 7fef04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:35.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.807+0000 7fef04ff9640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:38:35.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.807+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef4004eb0 con 0x7fef080ac5a0 2024-03-29T11:38:35.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 client.?.objecter ms_handle_connect 0x7fef080b0770 2024-03-29T11:38:35.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:35.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206071313 0 0) 0x7fef081eca20 con 0x7fef080b0770 2024-03-29T11:38:35.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:35.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:35.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:35.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:38:35.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef0807f810 con 0x7fef080b0770 2024-03-29T11:38:35.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 client.?.objecter ms_handle_connect 0x7fef0814a700 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984436443 0 0) 0x7fef0807f810 con 0x7fef080b0770 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef081eca20 con 0x7fef080b0770 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7feefc0017d0 con 0x7fef080b0770 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:35.814 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016242601 0 0) 0x7fef081ed5c0 con 0x7fef0814a700 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.809+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef0807f810 con 0x7fef0814a700 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3643713337 0 0) 0x7fef081eca20 con 0x7fef080b0770 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 >> v1:172.21.15.22:6790/0 conn(0x7fef080ac5a0 legacy=0x7fef0807e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:35.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 >> v1:172.21.15.204:6789/0 conn(0x7fef0814a700 legacy=0x7fef081f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 1 monclient: found mon.a 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef081ec6b0 con 0x7fef080b0770 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 10 monclient: _renew_subs 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0807d420 con 0x7fef080b0770 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:35.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 10 monclient: _renew_subs 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0807b1d0 con 0x7fef080b0770 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef0e625640 1 librados: init done 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feefc001ce0 con 0x7fef080b0770 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7feefc003790 con 0x7fef080b0770 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef04ff9640 10 monclient: dump: 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:35.817 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:35.818 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:35.818 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:35.818 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:35.818 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:35.818 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:35.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.810+0000 7fef05ffb640 4 set_mon_vals no callback set 2024-03-29T11:38:35.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.813+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7feefc019760 con 0x7fef080b0770 2024-03-29T11:38:35.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.813+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7feefc0837c0 con 0x7fef080b0770 2024-03-29T11:38:35.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.813+0000 7fef04ff9640 10 client.4377.objecter ms_dispatch 0x7fef08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:35.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.813+0000 7fef04ff9640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:35.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.813+0000 7fef04ff9640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:35.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.813+0000 7fef04ff9640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T11:38:35.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.814+0000 7fef07fff640 10 monclient: get_auth_request con 0x7feef406de10 auth_method 0 2024-03-29T11:38:35.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.814+0000 7fef0e625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:35.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.814+0000 7fef0e625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:35.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.814+0000 7fef0e625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:38:35.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.814+0000 7fef04ff9640 10 client.4377.objecter ms_handle_connect 0x7feef406de10 2024-03-29T11:38:35.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.814+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee8005080 con 0x7fef080b0770 2024-03-29T11:38:35.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.825+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feefc04eed0 con 0x7fef080b0770 2024-03-29T11:38:35.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.825+0000 7fef04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:35.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:35.825+0000 7fef04ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:36.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.156+0000 7fef0e625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.156+0000 7fef0e625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.157+0000 7fef0e625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:38:36.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.157+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feee8005330 con 0x7fef080b0770 2024-03-29T11:38:36.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.159+0000 7fef04ff9640 1 -- 172.21.15.22:0/2759703081 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7feefc04eed0 con 0x7fef080b0770 2024-03-29T11:38:36.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.159+0000 7fef04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.159+0000 7fef04ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:36.164 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:38:36.164 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:36.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.164+0000 7fef0e625640 10 librados: watch_flush enter 2024-03-29T11:38:36.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.164+0000 7fef0e625640 10 librados: watch_flush exit 2024-03-29T11:38:36.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.164+0000 7fef0e625640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T11:38:36.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.164+0000 7fef0e625640 10 client.4377.objecter successfully canceled tick 2024-03-29T11:38:36.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.166+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 >> v1:172.21.15.204:6816/3848099522 conn(0x7feef406de10 legacy=0x7feef40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.166+0000 7fef0e625640 10 monclient: shutdown 2024-03-29T11:38:36.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.166+0000 7fef0e625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:36.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.166+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 >> v1:172.21.15.22:6789/0 conn(0x7fef080b0770 legacy=0x7fef081eceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.167+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 shutdown_connections 2024-03-29T11:38:36.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.167+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 >> 172.21.15.22:0/2759703081 conn(0x7fef0806b660 msgr2=0x7fef08071500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:36.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.170+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 shutdown_connections 2024-03-29T11:38:36.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.170+0000 7fef0e625640 1 -- 172.21.15.22:0/2759703081 wait complete. 2024-03-29T11:38:36.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:36.170+0000 7fef0e625640 1 librados: shutdown 2024-03-29T11:38:36.189 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:42.995 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:43.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.149+0000 7fb11defb640 10 monclient: get_monmap_and_config 2024-03-29T11:38:43.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.149+0000 7fb11defb640 10 monclient: build_initial_monmap 2024-03-29T11:38:43.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.149+0000 7fb11defb640 10 monclient: monmap: 2024-03-29T11:38:43.150 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:38:43.150819+0000 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:38:43.150819+0000 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:43.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.150+0000 7fb11defb640 1 Processor -- start 2024-03-29T11:38:43.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.150+0000 7fb11defb640 1 -- start start 2024-03-29T11:38:43.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.150+0000 7fb11defb640 10 monclient: init 2024-03-29T11:38:43.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:43.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 10 monclient(hunting): picked mon.noname-b con 0x7fb1181457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:43.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 10 monclient(hunting): picked mon.noname-c con 0x7fb11813f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:43.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 10 monclient(hunting): picked mon.noname-a con 0x7fb1180a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:43.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1180aaef0 con 0x7fb1180a9d70 2024-03-29T11:38:43.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118145bd0 con 0x7fb11813f650 2024-03-29T11:38:43.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118145eb0 con 0x7fb1181457f0 2024-03-29T11:38:43.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.151+0000 7fb11defb640 10 monclient(hunting): authenticate will time out at 1173.428709s 2024-03-29T11:38:43.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.155+0000 7fb1177fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb1181457f0 0x7fb11813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:46395/0 (socket says 172.21.15.22:46395) 2024-03-29T11:38:43.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.155+0000 7fb1177fe640 1 -- 172.21.15.22:0/3292489902 learned_addr learned my addr 172.21.15.22:0/3292489902 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070236705 0 0) 0x7fb1180aaef0 con 0x7fb1180a9d70 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb104003520 con 0x7fb1180a9d70 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213003758 0 0) 0x7fb118145eb0 con 0x7fb1181457f0 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1180aaef0 con 0x7fb1181457f0 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.156+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555943117 0 0) 0x7fb1180aaef0 con 0x7fb1181457f0 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 >> v1:172.21.15.204:6789/0 conn(0x7fb11813f650 legacy=0x7fb1180b2620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 >> v1:172.21.15.22:6789/0 conn(0x7fb1180a9d70 legacy=0x7fb11814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 1 monclient: found mon.noname-b 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118146340 con 0x7fb1181457f0 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb108001630 con 0x7fb1181457f0 2024-03-29T11:38:43.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: dump: 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:43.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb108001880 con 0x7fb1181457f0 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 1 -- 172.21.15.22:0/3292489902 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb108001630 con 0x7fb1181457f0 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.157+0000 7fb1167fc640 10 monclient: dump: 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:43.167 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:43.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.158+0000 7fb11defb640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:38:43.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.158+0000 7fb11defb640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:43.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.158+0000 7fb11defb640 4 set_mon_vals no callback set 2024-03-29T11:38:43.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.159+0000 7fb11defb640 10 monclient: shutdown 2024-03-29T11:38:43.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.159+0000 7fb11defb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:43.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.159+0000 7fb11defb640 1 -- 172.21.15.22:0/3292489902 >> v1:172.21.15.22:6790/0 conn(0x7fb1181457f0 legacy=0x7fb11813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.164+0000 7fb11defb640 1 -- 172.21.15.22:0/3292489902 shutdown_connections 2024-03-29T11:38:43.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.164+0000 7fb11defb640 1 -- 172.21.15.22:0/3292489902 >> 172.21.15.22:0/3292489902 conn(0x7fb118066a00 msgr2=0x7fb118066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:43.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.165+0000 7fb11defb640 1 -- 172.21.15.22:0/3292489902 shutdown_connections 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.165+0000 7fb11defb640 1 -- 172.21.15.22:0/3292489902 wait complete. 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.166+0000 7fb11defb640 10 monclient: build_initial_monmap 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.166+0000 7fb11defb640 10 monclient: monmap: 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:43.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.166+0000 7fb11defb640 1 librados: starting msgr at 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.166+0000 7fb11defb640 1 librados: starting objecter 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.166+0000 7fb11defb640 1 Processor -- start 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.167+0000 7fb11defb640 1 -- start start 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.167+0000 7fb11defb640 1 librados: setting wanted keys 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.167+0000 7fb11defb640 1 librados: calling monclient init 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.167+0000 7fb11defb640 10 monclient: init 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 10 monclient(hunting): picked mon.noname-b con 0x7fb1180a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 10 monclient(hunting): picked mon.noname-c con 0x7fb11813f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 10 monclient(hunting): picked mon.noname-a con 0x7fb1181457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118139ed0 con 0x7fb1181457f0 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb118132940 con 0x7fb1180a9d70 2024-03-29T11:38:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1181397d0 con 0x7fb11813f650 2024-03-29T11:38:43.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.168+0000 7fb11defb640 10 monclient(hunting): authenticate will time out at 1173.445466s 2024-03-29T11:38:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb1177fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fb1180a9d70 0x7fb118143320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:43951/0 (socket says 172.21.15.22:43951) 2024-03-29T11:38:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb1177fe640 1 -- 172.21.15.22:0/386033379 learned_addr learned my addr 172.21.15.22:0/386033379 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb10ffff640 10 client.?.objecter ms_handle_connect 0x7fb1180a9d70 2024-03-29T11:38:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb10ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987861924 0 0) 0x7fb118132940 con 0x7fb1180a9d70 2024-03-29T11:38:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb10ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb10ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb10ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb10ffff640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.173+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f8004eb0 con 0x7fb1180a9d70 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 576471924 0 0) 0x7fb0f8004eb0 con 0x7fb1180a9d70 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0f8005300 con 0x7fb1180a9d70 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb108001630 con 0x7fb1180a9d70 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 10 monclient(hunting): dump: 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:43.180 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3316311486 0 0) 0x7fb0f8005300 con 0x7fb1180a9d70 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 >> v1:172.21.15.22:6790/0 conn(0x7fb11813f650 legacy=0x7fb118131b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 >> v1:172.21.15.22:6789/0 conn(0x7fb1181457f0 legacy=0x7fb118132230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 1 monclient: found mon.b 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118139ab0 con 0x7fb1180a9d70 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.175+0000 7fb10ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.176+0000 7fb11defb640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.176+0000 7fb11defb640 10 monclient: _renew_subs 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.176+0000 7fb11defb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.176+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb118145280 con 0x7fb1180a9d70 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.177+0000 7fb11defb640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.177+0000 7fb11defb640 10 monclient: _renew_subs 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.177+0000 7fb11defb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.177+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb118144c30 con 0x7fb1180a9d70 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.177+0000 7fb11defb640 1 librados: init done 2024-03-29T11:38:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb11defb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb11defb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb11defb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb118145280 con 0x7fb1180a9d70 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb108004ad0 con 0x7fb1180a9d70 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb10ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb108001630 con 0x7fb1180a9d70 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb10ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb10ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.178+0000 7fb10ffff640 10 monclient: dump: 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:43.183 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:43.184 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:43.184 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:43.184 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:43.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.179+0000 7fb115ffb640 4 set_mon_vals no callback set 2024-03-29T11:38:43.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.180+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fb108004dc0 con 0x7fb1180a9d70 2024-03-29T11:38:43.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.181+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fb1080849c0 con 0x7fb1180a9d70 2024-03-29T11:38:43.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.181+0000 7fb10ffff640 10 client.4357.objecter ms_dispatch 0x7fb118000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.181+0000 7fb10ffff640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.181+0000 7fb10ffff640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.181+0000 7fb10ffff640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T11:38:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.182+0000 7fb116ffd640 10 monclient: get_auth_request con 0x7fb0f806e0e0 auth_method 0 2024-03-29T11:38:43.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.185+0000 7fb10ffff640 10 client.4357.objecter ms_handle_connect 0x7fb0f806e0e0 2024-03-29T11:38:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.186+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1080502a0 con 0x7fb1180a9d70 2024-03-29T11:38:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.186+0000 7fb10ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.186+0000 7fb10ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.559+0000 7fb11defb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.559+0000 7fb11defb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.559+0000 7fb11defb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.559+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb118144c30 con 0x7fb1180a9d70 2024-03-29T11:38:43.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.561+0000 7fb10ffff640 1 -- 172.21.15.22:0/386033379 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fb1080521d0 con 0x7fb1180a9d70 2024-03-29T11:38:43.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.561+0000 7fb10ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.561+0000 7fb10ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:43.562 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:38:43.563 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:43.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.564+0000 7fb11defb640 10 librados: watch_flush enter 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.564+0000 7fb11defb640 10 librados: watch_flush exit 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.564+0000 7fb11defb640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.564+0000 7fb11defb640 10 client.4357.objecter successfully canceled tick 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.564+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 >> v1:172.21.15.204:6816/3848099522 conn(0x7fb0f806e0e0 legacy=0x7fb0f80705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.565+0000 7fb11defb640 10 monclient: shutdown 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.565+0000 7fb11defb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.565+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 >> v1:172.21.15.204:6789/0 conn(0x7fb1180a9d70 legacy=0x7fb118143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.565+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 shutdown_connections 2024-03-29T11:38:43.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.565+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 >> 172.21.15.22:0/386033379 conn(0x7fb118066a00 msgr2=0x7fb11813a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.565+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 shutdown_connections 2024-03-29T11:38:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.565+0000 7fb11defb640 1 -- 172.21.15.22:0/386033379 wait complete. 2024-03-29T11:38:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:43.565+0000 7fb11defb640 1 librados: shutdown 2024-03-29T11:38:43.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:50.388 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.549+0000 7fc202ba4640 10 monclient: get_monmap_and_config 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.549+0000 7fc202ba4640 10 monclient: build_initial_monmap 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.549+0000 7fc202ba4640 10 monclient: monmap: 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:38:50.550870+0000 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:38:50.550870+0000 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:50.552 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:38:50.553 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:38:50.553 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:50.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.550+0000 7fc202ba4640 1 Processor -- start 2024-03-29T11:38:50.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.551+0000 7fc202ba4640 1 -- start start 2024-03-29T11:38:50.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.551+0000 7fc202ba4640 10 monclient: init 2024-03-29T11:38:50.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.551+0000 7fc202ba4640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:50.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:50.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 10 monclient(hunting): picked mon.noname-a con 0x7fc1fc0a9750 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:50.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 10 monclient(hunting): picked mon.noname-c con 0x7fc1fc0adca0 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:50.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 10 monclient(hunting): picked mon.noname-b con 0x7fc1fc146a80 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:50.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc0abdc0 con 0x7fc1fc0a9750 2024-03-29T11:38:50.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc0b0070 con 0x7fc1fc0adca0 2024-03-29T11:38:50.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc0af880 con 0x7fc1fc146a80 2024-03-29T11:38:50.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:50.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc202ba4640 10 monclient(hunting): authenticate will time out at 1180.829112s 2024-03-29T11:38:50.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc2023a3640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc1fc146a80 0x7fc1fc151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49128/0 (socket says 172.21.15.22:49128) 2024-03-29T11:38:50.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.552+0000 7fc2023a3640 1 -- 172.21.15.22:0/4239945651 learned_addr learned my addr 172.21.15.22:0/4239945651 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.555+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869611263 0 0) 0x7fc1fc0af880 con 0x7fc1fc146a80 2024-03-29T11:38:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.555+0000 7fc200ba0640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.555+0000 7fc200ba0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.555+0000 7fc200ba0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.555+0000 7fc200ba0640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:38:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.555+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e8003520 con 0x7fc1fc146a80 2024-03-29T11:38:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679314505 0 0) 0x7fc1fc0abdc0 con 0x7fc1fc0a9750 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc0af880 con 0x7fc1fc0a9750 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473936302 0 0) 0x7fc1fc0b0070 con 0x7fc1fc0adca0 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc0abdc0 con 0x7fc1fc0adca0 2024-03-29T11:38:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387356179 0 0) 0x7fc1e8003520 con 0x7fc1fc146a80 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 >> v1:172.21.15.204:6789/0 conn(0x7fc1fc0adca0 legacy=0x7fc1fc146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 >> v1:172.21.15.22:6789/0 conn(0x7fc1fc0a9750 legacy=0x7fc1fc143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient: _finish_hunting 0 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 monclient: found mon.noname-b 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc0ac030 con 0x7fc1fc146a80 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient: _finish_auth 0 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient: _check_auth_tickets 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc202ba4640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc202ba4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc1f4001990 con 0x7fc1fc146a80 2024-03-29T11:38:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.556+0000 7fc200ba0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc200ba0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc200ba0640 10 monclient: dump: 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc202ba4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1f4001af0 con 0x7fc1fc146a80 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc200ba0640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc202ba4640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:50.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc202ba4640 4 set_mon_vals no callback set 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc200ba0640 1 -- 172.21.15.22:0/4239945651 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc1f4001f40 con 0x7fc1fc146a80 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc200ba0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc200ba0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.557+0000 7fc200ba0640 10 monclient: dump: 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:50.562 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.558+0000 7fc202ba4640 10 monclient: shutdown 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.558+0000 7fc202ba4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.558+0000 7fc202ba4640 1 -- 172.21.15.22:0/4239945651 >> v1:172.21.15.22:6790/0 conn(0x7fc1fc146a80 legacy=0x7fc1fc151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.559+0000 7fc202ba4640 1 -- 172.21.15.22:0/4239945651 shutdown_connections 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.559+0000 7fc202ba4640 1 -- 172.21.15.22:0/4239945651 >> 172.21.15.22:0/4239945651 conn(0x7fc1fc0673e0 msgr2=0x7fc1fc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:50.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.560+0000 7fc202ba4640 1 -- 172.21.15.22:0/4239945651 shutdown_connections 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.560+0000 7fc202ba4640 1 -- 172.21.15.22:0/4239945651 wait complete. 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.561+0000 7fc202ba4640 10 monclient: build_initial_monmap 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.561+0000 7fc202ba4640 10 monclient: monmap: 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:50.564 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.562+0000 7fc202ba4640 1 librados: starting msgr at 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.562+0000 7fc202ba4640 1 librados: starting objecter 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.562+0000 7fc202ba4640 1 Processor -- start 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.562+0000 7fc202ba4640 1 -- start start 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.562+0000 7fc202ba4640 1 librados: setting wanted keys 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.562+0000 7fc202ba4640 1 librados: calling monclient init 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.562+0000 7fc202ba4640 10 monclient: init 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 10 monclient(hunting): picked mon.noname-a con 0x7fc1fc0a9750 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 10 monclient(hunting): picked mon.noname-c con 0x7fc1fc0adca0 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 10 monclient(hunting): picked mon.noname-b con 0x7fc1fc146a80 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc1db3d0 con 0x7fc1fc0a9750 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc1db6b0 con 0x7fc1fc146a80 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc124490 con 0x7fc1fc0adca0 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc202ba4640 10 monclient(hunting): authenticate will time out at 1180.840320s 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.563+0000 7fc2013a1640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc1fc0adca0 0x7fc1fc13e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49848/0 (socket says 172.21.15.22:49848) 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc2013a1640 1 -- 172.21.15.22:0/2145064479 learned_addr learned my addr 172.21.15.22:0/2145064479 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc1f27fc640 10 client.?.objecter ms_handle_connect 0x7fc1fc0adca0 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc1f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872136592 0 0) 0x7fc1fc124490 con 0x7fc1fc0adca0 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc1f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc1f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc1f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc1f27fc640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.564+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e0004eb0 con 0x7fc1fc0adca0 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.565+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527358686 0 0) 0x7fc1e0004eb0 con 0x7fc1fc0adca0 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.565+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1fc124490 con 0x7fc1fc0adca0 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.565+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc1ec001a40 con 0x7fc1fc0adca0 2024-03-29T11:38:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.565+0000 7fc1f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.565+0000 7fc1f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.565+0000 7fc1f27fc640 10 monclient(hunting): dump: 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:50.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.565+0000 7fc1f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 client.?.objecter ms_handle_connect 0x7fc1fc146a80 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915946831 0 0) 0x7fc1fc1db6b0 con 0x7fc1fc146a80 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:38:50.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e0004eb0 con 0x7fc1fc146a80 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4188416943 0 0) 0x7fc1fc124490 con 0x7fc1fc0adca0 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 >> v1:172.21.15.204:6789/0 conn(0x7fc1fc146a80 legacy=0x7fc1fc0716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 >> v1:172.21.15.22:6789/0 conn(0x7fc1fc0a9750 legacy=0x7fc1fc1dbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 1 monclient: found mon.c 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc1dc250 con 0x7fc1fc0adca0 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.568+0000 7fc1f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:38:50.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 10 monclient: _renew_subs 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc13e880 con 0x7fc1fc0adca0 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 10 monclient: _renew_subs 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc06e550 con 0x7fc1fc0adca0 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc202ba4640 1 librados: init done 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1ec001d40 con 0x7fc1fc0adca0 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc1f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.569+0000 7fc1f3fff640 4 set_mon_vals no callback set 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.571+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc1ec003a60 con 0x7fc1fc0adca0 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.571+0000 7fc1f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.571+0000 7fc1f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.571+0000 7fc1f27fc640 10 monclient: dump: 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:50.573 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:50.574 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:50.574 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:50.574 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.574 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:50.574 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:50.574 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:50.574 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:50.574 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:50.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.571+0000 7fc202ba4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.571+0000 7fc202ba4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.571+0000 7fc202ba4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:50.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.571+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e4005080 con 0x7fc1fc0adca0 2024-03-29T11:38:50.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.572+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fc1ec003ca0 con 0x7fc1fc0adca0 2024-03-29T11:38:50.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.572+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fc1ec084c20 con 0x7fc1fc0adca0 2024-03-29T11:38:50.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.572+0000 7fc1f27fc640 10 client.4388.objecter ms_dispatch 0x7fc1fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:50.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.572+0000 7fc1f27fc640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:50.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.572+0000 7fc1f27fc640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:50.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.572+0000 7fc1f27fc640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:38:50.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.576+0000 7fc201ba2640 10 monclient: get_auth_request con 0x7fc1e006dc60 auth_method 0 2024-03-29T11:38:50.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.577+0000 7fc1f27fc640 10 client.4388.objecter ms_handle_connect 0x7fc1e006dc60 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.580+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1ec050500 con 0x7fc1fc0adca0 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.581+0000 7fc1f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.581+0000 7fc1f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:50.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.906+0000 7fc202ba4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:50.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.906+0000 7fc202ba4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.906+0000 7fc202ba4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:38:50.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.906+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1e4005330 con 0x7fc1fc0adca0 2024-03-29T11:38:50.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.907+0000 7fc1f27fc640 1 -- 172.21.15.22:0/2145064479 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fc1ec052430 con 0x7fc1fc0adca0 2024-03-29T11:38:50.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.907+0000 7fc1f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:50.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.907+0000 7fc1f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:50.910 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:38:50.910 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:50.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.912+0000 7fc202ba4640 10 librados: watch_flush enter 2024-03-29T11:38:50.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 10 librados: watch_flush exit 2024-03-29T11:38:50.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:38:50.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:38:50.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 >> v1:172.21.15.204:6816/3848099522 conn(0x7fc1e006dc60 legacy=0x7fc1e0070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 10 monclient: shutdown 2024-03-29T11:38:50.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:50.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 >> v1:172.21.15.22:6790/0 conn(0x7fc1fc0adca0 legacy=0x7fc1fc13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 shutdown_connections 2024-03-29T11:38:50.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 >> 172.21.15.22:0/2145064479 conn(0x7fc1fc0673e0 msgr2=0x7fc1fc0aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:50.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 shutdown_connections 2024-03-29T11:38:50.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.913+0000 7fc202ba4640 1 -- 172.21.15.22:0/2145064479 wait complete. 2024-03-29T11:38:50.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:50.914+0000 7fc202ba4640 1 librados: shutdown 2024-03-29T11:38:50.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:57.738 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.896+0000 7f064b9c8640 10 monclient: get_monmap_and_config 2024-03-29T11:38:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.896+0000 7f064b9c8640 10 monclient: build_initial_monmap 2024-03-29T11:38:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.896+0000 7f064b9c8640 10 monclient: monmap: 2024-03-29T11:38:57.898 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:57.898 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:57.898 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:38:57.897969+0000 2024-03-29T11:38:57.898 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:38:57.897969+0000 2024-03-29T11:38:57.898 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:57.899 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:57.899 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:57.899 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:38:57.899 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:38:57.899 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.897+0000 7f064b9c8640 1 Processor -- start 2024-03-29T11:38:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.897+0000 7f064b9c8640 1 -- start start 2024-03-29T11:38:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.897+0000 7f064b9c8640 10 monclient: init 2024-03-29T11:38:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f06440b0770 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f064414a700 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f06440ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06440aba70 con 0x7f064414a700 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06440aecb0 con 0x7f06440b0770 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06440af150 con 0x7f06440ac5a0 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.898+0000 7f064b9c8640 10 monclient(hunting): authenticate will time out at 1188.175905s 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.899+0000 7f0649f65640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f06440ac5a0 0x7f0644151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58480/0 (socket says 172.21.15.22:58480) 2024-03-29T11:38:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.899+0000 7f0649f65640 1 -- 172.21.15.22:0/2609817668 learned_addr learned my addr 172.21.15.22:0/2609817668 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.902+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927445574 0 0) 0x7f06440af150 con 0x7f06440ac5a0 2024-03-29T11:38:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.902+0000 7f0637fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.902+0000 7f0637fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.902+0000 7f0637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.902+0000 7f0637fff640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:38:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.902+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0630003520 con 0x7f06440ac5a0 2024-03-29T11:38:57.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3487751980 0 0) 0x7f0630003520 con 0x7f06440ac5a0 2024-03-29T11:38:57.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 >> v1:172.21.15.204:6789/0 conn(0x7f06440b0770 legacy=0x7f0644149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 >> v1:172.21.15.22:6789/0 conn(0x7f064414a700 legacy=0x7f06440abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 1 monclient: found mon.noname-b 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06440ac9d0 con 0x7f06440ac5a0 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f063c001690 con 0x7f06440ac5a0 2024-03-29T11:38:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f064b9c8640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f064b9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.904+0000 7f0637fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.905+0000 7f0637fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.905+0000 7f0637fff640 10 monclient: dump: 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.905+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f063c0018b0 con 0x7f06440ac5a0 2024-03-29T11:38:57.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.905+0000 7f064b9c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:57.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.905+0000 7f0637fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:57.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.905+0000 7f0637fff640 1 -- 172.21.15.22:0/2609817668 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f063c001c40 con 0x7f06440ac5a0 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.905+0000 7f064b9c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.905+0000 7f064b9c8640 4 set_mon_vals no callback set 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.906+0000 7f064b9c8640 10 monclient: shutdown 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.906+0000 7f064b9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.906+0000 7f064b9c8640 1 -- 172.21.15.22:0/2609817668 >> v1:172.21.15.22:6790/0 conn(0x7f06440ac5a0 legacy=0x7f0644151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.906+0000 7f0637fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.906+0000 7f064b9c8640 1 -- 172.21.15.22:0/2609817668 shutdown_connections 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.906+0000 7f064b9c8640 1 -- 172.21.15.22:0/2609817668 >> 172.21.15.22:0/2609817668 conn(0x7f064406b660 msgr2=0x7f064406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.907+0000 7f064b9c8640 1 -- 172.21.15.22:0/2609817668 shutdown_connections 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.907+0000 7f064b9c8640 1 -- 172.21.15.22:0/2609817668 wait complete. 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.908+0000 7f064b9c8640 10 monclient: build_initial_monmap 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.908+0000 7f064b9c8640 10 monclient: monmap: 2024-03-29T11:38:57.911 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.908+0000 7f064b9c8640 1 librados: starting msgr at 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.908+0000 7f064b9c8640 1 librados: starting objecter 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.908+0000 7f064b9c8640 1 Processor -- start 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.909+0000 7f064b9c8640 1 -- start start 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.909+0000 7f064b9c8640 1 librados: setting wanted keys 2024-03-29T11:38:57.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.909+0000 7f064b9c8640 1 librados: calling monclient init 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.909+0000 7f064b9c8640 10 monclient: init 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.910+0000 7f064b9c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.910+0000 7f064b9c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.910+0000 7f064b9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f06440b0770 addr v1:172.21.15.22:6790/0 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.910+0000 7f064b9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f064414a700 addr v1:172.21.15.22:6789/0 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.913+0000 7f0648f63640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f064414a700 0x7f0644144da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:37418/0 (socket says 172.21.15.22:37418) 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.913+0000 7f064b9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f06441454b0 addr v1:172.21.15.204:6789/0 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.913+0000 7f064b9c8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06441239d0 con 0x7f064414a700 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f064b9c8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0644127ed0 con 0x7f06441454b0 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f064b9c8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06441273e0 con 0x7f06440b0770 2024-03-29T11:38:57.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f064b9c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f064b9c8640 10 monclient(hunting): authenticate will time out at 1188.191050s 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f0649f65640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f06441454b0 0x7f06441dba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:42623/0 (socket says 172.21.15.22:42623) 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f0649f65640 1 -- 172.21.15.22:0/3288931599 learned_addr learned my addr 172.21.15.22:0/3288931599 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f0635ffb640 10 client.?.objecter ms_handle_connect 0x7f06441454b0 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f0635ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f0635ffb640 10 client.?.objecter ms_handle_connect 0x7f064414a700 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.914+0000 7f0635ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573126096 0 0) 0x7f0644127ed0 con 0x7f06441454b0 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:57.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0628004eb0 con 0x7f06441454b0 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 10 client.?.objecter ms_handle_connect 0x7f06440b0770 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892475707 0 0) 0x7f06441239d0 con 0x7f064414a700 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.915+0000 7f0635ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:38:57.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0644127ed0 con 0x7f064414a700 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458874244 0 0) 0x7f06441273e0 con 0x7f06440b0770 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06441239d0 con 0x7f06440b0770 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2833807790 0 0) 0x7f0628004eb0 con 0x7f06441454b0 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06441273e0 con 0x7f06441454b0 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f063c001bc0 con 0x7f06441454b0 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.916+0000 7f0635ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:57.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3108849728 0 0) 0x7f0644127ed0 con 0x7f064414a700 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0628004eb0 con 0x7f064414a700 2024-03-29T11:38:57.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f06380017d0 con 0x7f064414a700 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:57.928 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3890627787 0 0) 0x7f06441273e0 con 0x7f06441454b0 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 >> v1:172.21.15.22:6790/0 conn(0x7f06440b0770 legacy=0x7f0644126010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 >> v1:172.21.15.22:6789/0 conn(0x7f064414a700 legacy=0x7f0644144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.917+0000 7f0635ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f0635ffb640 1 monclient: found mon.b 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f0635ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f0635ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0644128120 con 0x7f06441454b0 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f0635ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f0635ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f0635ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:57.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f064b9c8640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f064b9c8640 10 monclient: _renew_subs 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f064b9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.918+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0644126d70 con 0x7f06441454b0 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f064b9c8640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f064b9c8640 10 monclient: _renew_subs 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f064b9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06441dc150 con 0x7f06441454b0 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f064b9c8640 1 librados: init done 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f063c001d20 con 0x7f06441454b0 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f0635ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f063c004c30 con 0x7f06441454b0 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f06377fe640 4 set_mon_vals no callback set 2024-03-29T11:38:57.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f0635ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f0635ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.919+0000 7f0635ffb640 10 monclient: dump: 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.921+0000 7f064b9c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:57.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.921+0000 7f064b9c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:57.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.921+0000 7f064b9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:57.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.921+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0624005080 con 0x7f06441454b0 2024-03-29T11:38:57.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.922+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f063c01ac00 con 0x7f06441454b0 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.922+0000 7f0649764640 10 monclient: get_auth_request con 0x7f062806dd60 auth_method 0 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.923+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f063c0849f0 con 0x7f06441454b0 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.923+0000 7f0635ffb640 10 client.4369.objecter ms_dispatch 0x7f0644000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.923+0000 7f0635ffb640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.923+0000 7f0635ffb640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.923+0000 7f0635ffb640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.923+0000 7f0635ffb640 10 client.4369.objecter ms_handle_connect 0x7f062806dd60 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.928+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f063c0502d0 con 0x7f06441454b0 2024-03-29T11:38:57.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.929+0000 7f0635ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:57.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:57.929+0000 7f0635ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.334+0000 7f064b9c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.334+0000 7f064b9c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.334+0000 7f064b9c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:38:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.334+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0624005330 con 0x7f06441454b0 2024-03-29T11:38:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.335+0000 7f0635ffb640 1 -- 172.21.15.22:0/3288931599 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f063c052200 con 0x7f06441454b0 2024-03-29T11:38:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.335+0000 7f0635ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.335+0000 7f0635ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:58.338 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:38:58.338 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:58.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.341+0000 7f064b9c8640 10 librados: watch_flush enter 2024-03-29T11:38:58.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.341+0000 7f064b9c8640 10 librados: watch_flush exit 2024-03-29T11:38:58.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.341+0000 7f064b9c8640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T11:38:58.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.341+0000 7f064b9c8640 10 client.4369.objecter successfully canceled tick 2024-03-29T11:38:58.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.342+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 >> v1:172.21.15.204:6816/3848099522 conn(0x7f062806dd60 legacy=0x7f0628070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.342+0000 7f064b9c8640 10 monclient: shutdown 2024-03-29T11:38:58.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.342+0000 7f064b9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:58.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.342+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 >> v1:172.21.15.204:6789/0 conn(0x7f06441454b0 legacy=0x7f06441dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.342+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 shutdown_connections 2024-03-29T11:38:58.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.342+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 >> 172.21.15.22:0/3288931599 conn(0x7f064406b660 msgr2=0x7f06441491f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:58.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.343+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 shutdown_connections 2024-03-29T11:38:58.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.343+0000 7f064b9c8640 1 -- 172.21.15.22:0/3288931599 wait complete. 2024-03-29T11:38:58.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:38:58.343+0000 7f064b9c8640 1 librados: shutdown 2024-03-29T11:38:58.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:05.168 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:05.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.315+0000 7f35980b5640 10 monclient: get_monmap_and_config 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.315+0000 7f35980b5640 10 monclient: build_initial_monmap 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.315+0000 7f35980b5640 10 monclient: monmap: 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:39:05.316796+0000 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:39:05.316796+0000 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:39:05.317 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:05.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.316+0000 7f35980b5640 1 Processor -- start 2024-03-29T11:39:05.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.316+0000 7f35980b5640 1 -- start start 2024-03-29T11:39:05.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.316+0000 7f35980b5640 10 monclient: init 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.317+0000 7f35980b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.317+0000 7f35980b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.317+0000 7f35980b5640 10 monclient(hunting): picked mon.noname-c con 0x7f35900b4890 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.317+0000 7f3595e51640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f35900b4890 0x7f359013b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:44240/0 (socket says 172.21.15.22:44240) 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3595e51640 1 -- 172.21.15.22:0/1834044014 learned_addr learned my addr 172.21.15.22:0/1834044014 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f35980b5640 10 monclient(hunting): picked mon.noname-b con 0x7f359013c040 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f35980b5640 10 monclient(hunting): picked mon.noname-a con 0x7f35901427e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f35980b5640 1 -- 172.21.15.22:0/1834044014 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900b42f0 con 0x7f35901427e0 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f35980b5640 1 -- 172.21.15.22:0/1834044014 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900aa7a0 con 0x7f35900b4890 2024-03-29T11:39:05.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f35980b5640 1 -- 172.21.15.22:0/1834044014 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900aaa10 con 0x7f359013c040 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f35980b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f35980b5640 10 monclient(hunting): authenticate will time out at 1195.595553s 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574516045 0 0) 0x7f35900aaa10 con 0x7f359013c040 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f357c003520 con 0x7f359013c040 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88026341 0 0) 0x7f35900b42f0 con 0x7f35901427e0 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:39:05.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.318+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35900aaa10 con 0x7f35901427e0 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3383994876 0 0) 0x7f357c003520 con 0x7f359013c040 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 >> v1:172.21.15.204:6789/0 conn(0x7f35900b4890 legacy=0x7f359013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 >> v1:172.21.15.22:6789/0 conn(0x7f35901427e0 legacy=0x7f359014d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 10 monclient: _finish_hunting 0 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 1 monclient: found mon.noname-b 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900aacf0 con 0x7f359013c040 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 10 monclient: _finish_auth 0 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 10 monclient: _check_auth_tickets 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f35800015c0 con 0x7f359013c040 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f3594e4f640 10 monclient: dump: 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f35980b5640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.319+0000 7f35980b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:05.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.320+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35800017e0 con 0x7f359013c040 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.320+0000 7f3594e4f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.320+0000 7f3594e4f640 1 -- 172.21.15.22:0/1834044014 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3580001af0 con 0x7f359013c040 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.320+0000 7f3594e4f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.320+0000 7f3594e4f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.320+0000 7f3594e4f640 10 monclient: dump: 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:05.327 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.320+0000 7f35980b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.320+0000 7f35980b5640 4 set_mon_vals no callback set 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.321+0000 7f35980b5640 10 monclient: shutdown 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.321+0000 7f35980b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.321+0000 7f35980b5640 1 -- 172.21.15.22:0/1834044014 >> v1:172.21.15.22:6790/0 conn(0x7f359013c040 legacy=0x7f35901420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.321+0000 7f35980b5640 1 -- 172.21.15.22:0/1834044014 shutdown_connections 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.321+0000 7f35980b5640 1 -- 172.21.15.22:0/1834044014 >> 172.21.15.22:0/1834044014 conn(0x7f359006d180 msgr2=0x7f3590066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.321+0000 7f35980b5640 1 -- 172.21.15.22:0/1834044014 shutdown_connections 2024-03-29T11:39:05.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.321+0000 7f35980b5640 1 -- 172.21.15.22:0/1834044014 wait complete. 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.322+0000 7f35980b5640 10 monclient: build_initial_monmap 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.322+0000 7f35980b5640 10 monclient: monmap: 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:05.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.323+0000 7f35980b5640 1 librados: starting msgr at 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.323+0000 7f35980b5640 1 librados: starting objecter 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.323+0000 7f35980b5640 1 Processor -- start 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.323+0000 7f35980b5640 1 -- start start 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.323+0000 7f35980b5640 1 librados: setting wanted keys 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.323+0000 7f35980b5640 1 librados: calling monclient init 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.323+0000 7f35980b5640 10 monclient: init 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.324+0000 7f35980b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.324+0000 7f35980b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.324+0000 7f35980b5640 10 monclient(hunting): picked mon.noname-c con 0x7f35900b4890 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.324+0000 7f35980b5640 10 monclient(hunting): picked mon.noname-b con 0x7f359013c040 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.324+0000 7f35980b5640 10 monclient(hunting): picked mon.noname-a con 0x7f35901427e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.324+0000 7f35980b5640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359013a730 con 0x7f35901427e0 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.324+0000 7f3596652640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f35901427e0 0x7f359013a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:41074/0 (socket says 172.21.15.22:41074) 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.324+0000 7f3596652640 1 -- 172.21.15.22:0/1732339601 learned_addr learned my addr 172.21.15.22:0/1732339601 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:05.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 client.?.objecter ms_handle_connect 0x7f35901427e0 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 client.?.objecter ms_handle_connect 0x7f35900b4890 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900aeec0 con 0x7f359013c040 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900ad9f0 con 0x7f35900b4890 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35980b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35980b5640 10 monclient(hunting): authenticate will time out at 1195.602260s 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961098833 0 0) 0x7f359013a730 con 0x7f35901427e0 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3574004eb0 con 0x7f35901427e0 2024-03-29T11:39:05.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682583234 0 0) 0x7f35900ad9f0 con 0x7f35900b4890 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f359013a730 con 0x7f35900b4890 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.325+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2401036197 0 0) 0x7f3574004eb0 con 0x7f35901427e0 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35900ad9f0 con 0x7f35901427e0 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3588001d10 con 0x7f35901427e0 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient(hunting): dump: 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:05.335 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 465219713 0 0) 0x7f359013a730 con 0x7f35900b4890 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3574004eb0 con 0x7f35900b4890 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f358c001780 con 0x7f35900b4890 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient(hunting): dump: 2024-03-29T11:39:05.336 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4161586631 0 0) 0x7f3574004eb0 con 0x7f35900b4890 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 >> v1:172.21.15.204:6789/0 conn(0x7f359013c040 legacy=0x7f3590139910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 >> v1:172.21.15.22:6789/0 conn(0x7f35901427e0 legacy=0x7f359013a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 monclient: found mon.c 2024-03-29T11:39:05.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900adf10 con 0x7f35900b4890 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 10 monclient: _renew_subs 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35900ad050 con 0x7f35900b4890 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 10 monclient: _renew_subs 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:05.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35901d7780 con 0x7f35900b4890 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35980b5640 1 librados: init done 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f358c001bb0 con 0x7f35900b4890 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f35867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.326+0000 7f3587fff640 4 set_mon_vals no callback set 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.327+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f358c004980 con 0x7f35900b4890 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.327+0000 7f35867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.327+0000 7f35867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.327+0000 7f35867fc640 10 monclient: dump: 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:05.339 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.328+0000 7f35980b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.328+0000 7f35980b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.328+0000 7f35980b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.328+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35900adb80 con 0x7f35900b4890 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.329+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f358c004b60 con 0x7f35900b4890 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.330+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f358c084920 con 0x7f35900b4890 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.330+0000 7f35867fc640 10 client.4400.objecter ms_dispatch 0x7f3590000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.330+0000 7f35867fc640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.330+0000 7f35867fc640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.330+0000 7f35867fc640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T11:39:05.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.330+0000 7f3595650640 10 monclient: get_auth_request con 0x7f357406de30 auth_method 0 2024-03-29T11:39:05.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.332+0000 7f35867fc640 10 client.4400.objecter ms_handle_connect 0x7f357406de30 2024-03-29T11:39:05.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.339+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f358c050260 con 0x7f35900b4890 2024-03-29T11:39:05.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.339+0000 7f35867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.339+0000 7f35867fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:05.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.709+0000 7f35980b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.709+0000 7f35980b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.709+0000 7f35980b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:05.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.709+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f359004f730 con 0x7f35900b4890 2024-03-29T11:39:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.710+0000 7f35867fc640 1 -- 172.21.15.22:0/1732339601 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f358c083fd0 con 0x7f35900b4890 2024-03-29T11:39:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.710+0000 7f35867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.710+0000 7f35867fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:05.711 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:39:05.711 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 10 librados: watch_flush enter 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 10 librados: watch_flush exit 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 10 client.4400.objecter successfully canceled tick 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 >> v1:172.21.15.204:6816/3848099522 conn(0x7f357406de30 legacy=0x7f35740702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 10 monclient: shutdown 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 >> v1:172.21.15.22:6790/0 conn(0x7f35900b4890 legacy=0x7f35901d6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 shutdown_connections 2024-03-29T11:39:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.713+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 >> 172.21.15.22:0/1732339601 conn(0x7f359006d180 msgr2=0x7f35900b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.714+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 shutdown_connections 2024-03-29T11:39:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.714+0000 7f35980b5640 1 -- 172.21.15.22:0/1732339601 wait complete. 2024-03-29T11:39:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:05.714+0000 7f35980b5640 1 librados: shutdown 2024-03-29T11:39:05.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:12.535 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:12.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.669+0000 7faee181e640 10 monclient: get_monmap_and_config 2024-03-29T11:39:12.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.669+0000 7faee181e640 10 monclient: build_initial_monmap 2024-03-29T11:39:12.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.669+0000 7faee181e640 10 monclient: monmap: 2024-03-29T11:39:12.671 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:39:12.670716+0000 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:39:12.670716+0000 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.670+0000 7faee181e640 1 Processor -- start 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.670+0000 7faee181e640 1 -- start start 2024-03-29T11:39:12.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.670+0000 7faee181e640 10 monclient: init 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 10 monclient(hunting): picked mon.noname-b con 0x7faedc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 10 monclient(hunting): picked mon.noname-a con 0x7faedc13f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 10 monclient(hunting): picked mon.noname-c con 0x7faedc0a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc0aaef0 con 0x7faedc13f650 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc145bd0 con 0x7faedc0a9d70 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc145eb0 con 0x7faedc1457f0 2024-03-29T11:39:12.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faee181e640 10 monclient(hunting): authenticate will time out at 1202.948694s 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faedaffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7faedc1457f0 0x7faedc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:32892/0 (socket says 172.21.15.22:32892) 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.671+0000 7faedaffd640 1 -- 172.21.15.22:0/3617238268 learned_addr learned my addr 172.21.15.22:0/3617238268 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226409372 0 0) 0x7faedc145eb0 con 0x7faedc1457f0 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec4003520 con 0x7faedc1457f0 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382609983 0 0) 0x7faedc145bd0 con 0x7faedc0a9d70 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:39:12.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec40048a0 con 0x7faedc0a9d70 2024-03-29T11:39:12.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3027686131 0 0) 0x7faec40048a0 con 0x7faedc0a9d70 2024-03-29T11:39:12.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 >> v1:172.21.15.22:6790/0 conn(0x7faedc1457f0 legacy=0x7faedc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.672+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 >> v1:172.21.15.22:6789/0 conn(0x7faedc13f650 legacy=0x7faedc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:12.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:12.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:39:12.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc146340 con 0x7faedc0a9d70 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faed40019b0 con 0x7faedc0a9d70 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faed9ffb640 10 monclient: dump: 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:12.679 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faee181e640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.674+0000 7faee181e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:12.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.675+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faed4001b80 con 0x7faedc0a9d70 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.675+0000 7faed9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.675+0000 7faee181e640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.675+0000 7faee181e640 4 set_mon_vals no callback set 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.675+0000 7faed9ffb640 1 -- 172.21.15.22:0/3617238268 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faed4001f10 con 0x7faedc0a9d70 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.675+0000 7faed9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.675+0000 7faed9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.675+0000 7faed9ffb640 10 monclient: dump: 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:12.681 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.676+0000 7faee181e640 10 monclient: shutdown 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.676+0000 7faee181e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.676+0000 7faee181e640 1 -- 172.21.15.22:0/3617238268 >> v1:172.21.15.204:6789/0 conn(0x7faedc0a9d70 legacy=0x7faedc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.676+0000 7faee181e640 1 -- 172.21.15.22:0/3617238268 shutdown_connections 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.676+0000 7faee181e640 1 -- 172.21.15.22:0/3617238268 >> 172.21.15.22:0/3617238268 conn(0x7faedc066a00 msgr2=0x7faedc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.676+0000 7faee181e640 1 -- 172.21.15.22:0/3617238268 shutdown_connections 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.676+0000 7faee181e640 1 -- 172.21.15.22:0/3617238268 wait complete. 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.677+0000 7faee181e640 10 monclient: build_initial_monmap 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.677+0000 7faee181e640 10 monclient: monmap: 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:12.683 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:12.684 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:39:12.684 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:39:12.684 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.678+0000 7faee181e640 1 librados: starting msgr at 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.678+0000 7faee181e640 1 librados: starting objecter 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.678+0000 7faee181e640 1 Processor -- start 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.678+0000 7faee181e640 1 -- start start 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.678+0000 7faee181e640 1 librados: setting wanted keys 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.678+0000 7faee181e640 1 librados: calling monclient init 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.678+0000 7faee181e640 10 monclient: init 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 10 monclient(hunting): picked mon.noname-c con 0x7faedc0a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:12.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 10 monclient(hunting): picked mon.noname-a con 0x7faedc13f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 10 monclient(hunting): picked mon.noname-b con 0x7faedc1457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc13ab70 con 0x7faedc13f650 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc13ae50 con 0x7faedc1457f0 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc13b130 con 0x7faedc0a9d70 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faee181e640 10 monclient(hunting): authenticate will time out at 1202.957625s 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faedaffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7faedc0a9d70 0x7faedc139630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35394/0 (socket says 172.21.15.22:35394) 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.680+0000 7faedaffd640 1 -- 172.21.15.22:0/1940254422 learned_addr learned my addr 172.21.15.22:0/1940254422 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 client.?.objecter ms_handle_connect 0x7faedc0a9d70 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 client.?.objecter ms_handle_connect 0x7faedc1457f0 2024-03-29T11:39:12.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024541798 0 0) 0x7faedc13b130 con 0x7faedc0a9d70 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec8004eb0 con 0x7faedc0a9d70 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047920577 0 0) 0x7faedc13ae50 con 0x7faedc1457f0 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faedc13b130 con 0x7faedc1457f0 2024-03-29T11:39:12.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3138275066 0 0) 0x7faedc13b130 con 0x7faedc1457f0 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faedc13ae50 con 0x7faedc1457f0 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faed4001f10 con 0x7faedc1457f0 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): dump: 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:12.694 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1518972284 0 0) 0x7faec8004eb0 con 0x7faedc0a9d70 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faedc13b130 con 0x7faedc0a9d70 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faecc001910 con 0x7faedc0a9d70 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.681+0000 7faec3fff640 10 monclient(hunting): dump: 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.695 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 275428785 0 0) 0x7faedc13b130 con 0x7faedc0a9d70 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 >> v1:172.21.15.204:6789/0 conn(0x7faedc1457f0 legacy=0x7faedc1e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 >> v1:172.21.15.22:6789/0 conn(0x7faedc13f650 legacy=0x7faedc134ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 1 monclient: found mon.c 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:12.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc139110 con 0x7faedc0a9d70 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faec3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 10 monclient: _renew_subs 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc140570 con 0x7faedc0a9d70 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 10 monclient: _renew_subs 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc138ea0 con 0x7faedc0a9d70 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.682+0000 7faee181e640 1 librados: init done 2024-03-29T11:39:12.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.684+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faecc0047e0 con 0x7faedc0a9d70 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.684+0000 7faec3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.684+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faecc004af0 con 0x7faedc0a9d70 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.684+0000 7faec3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.684+0000 7faec3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.684+0000 7faec3fff640 10 monclient: dump: 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:12.698 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.684+0000 7faed97fa640 4 set_mon_vals no callback set 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.685+0000 7faee181e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.685+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7faecc01ab00 con 0x7faedc0a9d70 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faee181e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faee181e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc139350 con 0x7faedc0a9d70 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7faecc084a60 con 0x7faedc0a9d70 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faec3fff640 10 client.4406.objecter ms_dispatch 0x7faedc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faec3fff640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faec3fff640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faeda7fc640 10 monclient: get_auth_request con 0x7faec806de80 auth_method 0 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.686+0000 7faec3fff640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:39:12.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.687+0000 7faec3fff640 10 client.4406.objecter ms_handle_connect 0x7faec806de80 2024-03-29T11:39:12.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.694+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faedc139350 con 0x7faedc0a9d70 2024-03-29T11:39:12.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.694+0000 7faec3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:12.694+0000 7faec3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:13.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.010+0000 7faee181e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.010+0000 7faee181e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.010+0000 7faee181e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:13.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.010+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faedc075f10 con 0x7faedc0a9d70 2024-03-29T11:39:13.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.010+0000 7faec3fff640 1 -- 172.21.15.22:0/1940254422 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7faecc050340 con 0x7faedc0a9d70 2024-03-29T11:39:13.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.010+0000 7faec3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.010+0000 7faec3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:13.013 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:39:13.013 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:13.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.014+0000 7faee181e640 10 librados: watch_flush enter 2024-03-29T11:39:13.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.014+0000 7faee181e640 10 librados: watch_flush exit 2024-03-29T11:39:13.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.014+0000 7faee181e640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:39:13.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.014+0000 7faee181e640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:39:13.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.014+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 >> v1:172.21.15.204:6816/3848099522 conn(0x7faec806de80 legacy=0x7faec8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:13.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.014+0000 7faee181e640 10 monclient: shutdown 2024-03-29T11:39:13.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.014+0000 7faee181e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:13.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.014+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 >> v1:172.21.15.22:6790/0 conn(0x7faedc0a9d70 legacy=0x7faedc139630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:13.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.015+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 shutdown_connections 2024-03-29T11:39:13.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.015+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 >> 172.21.15.22:0/1940254422 conn(0x7faedc066a00 msgr2=0x7faedc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:13.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.015+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 shutdown_connections 2024-03-29T11:39:13.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.015+0000 7faee181e640 1 -- 172.21.15.22:0/1940254422 wait complete. 2024-03-29T11:39:13.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:13.015+0000 7faee181e640 1 librados: shutdown 2024-03-29T11:39:13.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:19.837 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.983+0000 7f396092e640 10 monclient: get_monmap_and_config 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.983+0000 7f396092e640 10 monclient: build_initial_monmap 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.983+0000 7f396092e640 10 monclient: monmap: 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:19.989 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:39:19.985190+0000 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:39:19.985190+0000 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.987+0000 7f396092e640 1 Processor -- start 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.988+0000 7f396092e640 1 -- start start 2024-03-29T11:39:19.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.988+0000 7f396092e640 10 monclient: init 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.988+0000 7f396092e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.988+0000 7f396092e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.988+0000 7f396092e640 10 monclient(hunting): picked mon.noname-c con 0x7f3954034850 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.989+0000 7f396092e640 10 monclient(hunting): picked mon.noname-b con 0x7f39540c9550 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.989+0000 7f396092e640 10 monclient(hunting): picked mon.noname-a con 0x7f395402aac0 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.989+0000 7f396092e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395402c440 con 0x7f395402aac0 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.989+0000 7f396092e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395402aed0 con 0x7f3954034850 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.989+0000 7f396092e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395402b190 con 0x7f39540c9550 2024-03-29T11:39:19.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.989+0000 7f396092e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:19.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.989+0000 7f396092e640 10 monclient(hunting): authenticate will time out at 1210.266108s 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.992+0000 7f395bfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f395402aac0 0x7f39540cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:32927/0 (socket says 172.21.15.22:32927) 2024-03-29T11:39:19.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.992+0000 7f395bfff640 1 -- 172.21.15.22:0/1349850203 learned_addr learned my addr 172.21.15.22:0/1349850203 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.992+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816010850 0 0) 0x7f395402c440 con 0x7f395402aac0 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.992+0000 7f395a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.992+0000 7f395a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.992+0000 7f395a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:19.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.992+0000 7f395a7fc640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:39:19.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.992+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c003520 con 0x7f395402aac0 2024-03-29T11:39:19.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.995+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013111255 0 0) 0x7f394c003520 con 0x7f395402aac0 2024-03-29T11:39:19.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.995+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 >> v1:172.21.15.22:6790/0 conn(0x7f39540c9550 legacy=0x7f39540cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:19.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.995+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 >> v1:172.21.15.204:6789/0 conn(0x7f3954034850 legacy=0x7f39540c8e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.995+0000 7f395a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.995+0000 7f395a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.995+0000 7f395a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3954033060 con 0x7f395402aac0 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f395c04f730 con 0x7f395402aac0 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:19.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: dump: 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f396092e640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:39:20.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f396092e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f395c075f10 con 0x7f395402aac0 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 1 -- 172.21.15.22:0/1349850203 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f395c0758b0 con 0x7f395402aac0 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f395a7fc640 10 monclient: dump: 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.001 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f396092e640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.996+0000 7f396092e640 4 set_mon_vals no callback set 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.997+0000 7f396092e640 10 monclient: shutdown 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.997+0000 7f396092e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.997+0000 7f396092e640 1 -- 172.21.15.22:0/1349850203 >> v1:172.21.15.22:6789/0 conn(0x7f395402aac0 legacy=0x7f39540cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.997+0000 7f396092e640 1 -- 172.21.15.22:0/1349850203 shutdown_connections 2024-03-29T11:39:20.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.997+0000 7f396092e640 1 -- 172.21.15.22:0/1349850203 >> 172.21.15.22:0/1349850203 conn(0x7f39540058f0 msgr2=0x7f3954005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:20.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.998+0000 7f396092e640 1 -- 172.21.15.22:0/1349850203 shutdown_connections 2024-03-29T11:39:20.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.998+0000 7f396092e640 1 -- 172.21.15.22:0/1349850203 wait complete. 2024-03-29T11:39:20.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.999+0000 7f396092e640 10 monclient: build_initial_monmap 2024-03-29T11:39:20.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:19.999+0000 7f396092e640 10 monclient: monmap: 2024-03-29T11:39:20.004 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:20.004 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:20.004 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:39:20.004 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.000+0000 7f396092e640 1 librados: starting msgr at 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.000+0000 7f396092e640 1 librados: starting objecter 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.000+0000 7f396092e640 1 Processor -- start 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.000+0000 7f396092e640 1 -- start start 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.000+0000 7f396092e640 1 librados: setting wanted keys 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.000+0000 7f396092e640 1 librados: calling monclient init 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.000+0000 7f396092e640 10 monclient: init 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:20.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 10 monclient(hunting): picked mon.noname-c con 0x7f395402aac0 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:20.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 10 monclient(hunting): picked mon.noname-a con 0x7f3954034850 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:20.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 10 monclient(hunting): picked mon.noname-b con 0x7f39540c9550 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:20.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39540bfef0 con 0x7f3954034850 2024-03-29T11:39:20.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3954164d90 con 0x7f39540c9550 2024-03-29T11:39:20.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39540c1dd0 con 0x7f395402aac0 2024-03-29T11:39:20.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:20.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.002+0000 7f396092e640 10 monclient(hunting): authenticate will time out at 1210.279247s 2024-03-29T11:39:20.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.006+0000 7f395b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f395402aac0 0x7f39540c04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53545/0 (socket says 172.21.15.22:53545) 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f395b7fe640 1 -- 172.21.15.22:0/1712061336 learned_addr learned my addr 172.21.15.22:0/1712061336 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f3947fff640 10 client.?.objecter ms_handle_connect 0x7f395402aac0 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f3947fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.006+0000 7f395bfff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f39540c9550 0x7f39540b9660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:59305/0 (socket says 172.21.15.22:59305) 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337285044 0 0) 0x7f39540c1dd0 con 0x7f395402aac0 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f3947fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f3947fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f3947fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f3947fff640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.007+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3940004eb0 con 0x7f395402aac0 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 client.?.objecter ms_handle_connect 0x7f39540c9550 2024-03-29T11:39:20.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688245479 0 0) 0x7f3940004eb0 con 0x7f395402aac0 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39540c1dd0 con 0x7f395402aac0 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3950001870 con 0x7f395402aac0 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 monclient(hunting): dump: 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:20.017 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530198865 0 0) 0x7f3954164d90 con 0x7f39540c9550 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3940004eb0 con 0x7f39540c9550 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3054287296 0 0) 0x7f39540c1dd0 con 0x7f395402aac0 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 >> v1:172.21.15.204:6789/0 conn(0x7f39540c9550 legacy=0x7f39540b9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.008+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 >> v1:172.21.15.22:6789/0 conn(0x7f3954034850 legacy=0x7f3954164680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f3947fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:20.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f3947fff640 1 monclient: found mon.c 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f3947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f3947fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39540b9d70 con 0x7f395402aac0 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f3947fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f3947fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f3947fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 10 monclient: _renew_subs 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39540ba020 con 0x7f395402aac0 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 10 monclient: _renew_subs 2024-03-29T11:39:20.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:20.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39540bcd80 con 0x7f395402aac0 2024-03-29T11:39:20.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.012+0000 7f396092e640 1 librados: init done 2024-03-29T11:39:20.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.014+0000 7f396092e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.014+0000 7f396092e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.014+0000 7f396092e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:20.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.014+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3954004cd0 con 0x7f395402aac0 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.017+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39500019d0 con 0x7f395402aac0 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.017+0000 7f3947fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.017+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3950004aa0 con 0x7f395402aac0 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.017+0000 7f3947fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.017+0000 7f3947fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.017+0000 7f3947fff640 10 monclient: dump: 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:20.022 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.017+0000 7f3959ffb640 4 set_mon_vals no callback set 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.018+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f395001aa70 con 0x7f395402aac0 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.019+0000 7f395affd640 10 monclient: get_auth_request con 0x7f394006ddb0 auth_method 0 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.019+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f3950084a80 con 0x7f395402aac0 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.019+0000 7f3947fff640 10 client.4409.objecter ms_dispatch 0x7f395c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.020+0000 7f3947fff640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.020+0000 7f3947fff640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:20.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.020+0000 7f3947fff640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:39:20.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.020+0000 7f3947fff640 10 client.4409.objecter ms_handle_connect 0x7f394006ddb0 2024-03-29T11:39:20.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.024+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3950050190 con 0x7f395402aac0 2024-03-29T11:39:20.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.025+0000 7f3947fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.025+0000 7f3947fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:20.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.361+0000 7f396092e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.361+0000 7f396092e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.361+0000 7f396092e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.361+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39540086f0 con 0x7f395402aac0 2024-03-29T11:39:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.362+0000 7f3947fff640 1 -- 172.21.15.22:0/1712061336 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f3950052290 con 0x7f395402aac0 2024-03-29T11:39:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.362+0000 7f3947fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.362+0000 7f3947fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:20.364 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:39:20.364 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:20.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.366+0000 7f396092e640 10 librados: watch_flush enter 2024-03-29T11:39:20.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.366+0000 7f396092e640 10 librados: watch_flush exit 2024-03-29T11:39:20.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.366+0000 7f396092e640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:39:20.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.366+0000 7f396092e640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:39:20.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.366+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 >> v1:172.21.15.204:6816/3848099522 conn(0x7f394006ddb0 legacy=0x7f3940070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.366+0000 7f396092e640 10 monclient: shutdown 2024-03-29T11:39:20.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.366+0000 7f396092e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:20.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.366+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 >> v1:172.21.15.22:6790/0 conn(0x7f395402aac0 legacy=0x7f39540c04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.368+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 shutdown_connections 2024-03-29T11:39:20.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.368+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 >> 172.21.15.22:0/1712061336 conn(0x7f39540058f0 msgr2=0x7f39540043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:20.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.368+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 shutdown_connections 2024-03-29T11:39:20.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.368+0000 7f396092e640 1 -- 172.21.15.22:0/1712061336 wait complete. 2024-03-29T11:39:20.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:20.368+0000 7f396092e640 1 librados: shutdown 2024-03-29T11:39:20.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:27.194 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:27.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.341+0000 7f70f18b3640 10 monclient: get_monmap_and_config 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.341+0000 7f70f18b3640 10 monclient: build_initial_monmap 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.341+0000 7f70f18b3640 10 monclient: monmap: 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:39:27.343084+0000 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:39:27.343084+0000 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:27.344 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.342+0000 7f70f18b3640 1 Processor -- start 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.342+0000 7f70f18b3640 1 -- start start 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.342+0000 7f70f18b3640 10 monclient: init 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 10 monclient(hunting): picked mon.noname-b con 0x7f70ec0b4890 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:27.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 10 monclient(hunting): picked mon.noname-c con 0x7f70ec152120 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:27.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 10 monclient(hunting): picked mon.noname-a con 0x7f70ec071a60 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:27.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec06e7c0 con 0x7f70ec071a60 2024-03-29T11:39:27.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec06fa60 con 0x7f70ec152120 2024-03-29T11:39:27.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec06e2c0 con 0x7f70ec0b4890 2024-03-29T11:39:27.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:27.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.343+0000 7f70f18b3640 10 monclient(hunting): authenticate will time out at 1217.620789s 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.346+0000 7f70eb7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f70ec071a60 0x7f70ec15a4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34723/0 (socket says 172.21.15.22:34723) 2024-03-29T11:39:27.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.346+0000 7f70eb7fe640 1 -- 172.21.15.22:0/1164227279 learned_addr learned my addr 172.21.15.22:0/1164227279 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307462039 0 0) 0x7f70ec06e7c0 con 0x7f70ec071a60 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d4003520 con 0x7f70ec071a60 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076943604 0 0) 0x7f70ec06e2c0 con 0x7f70ec0b4890 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ec06e7c0 con 0x7f70ec0b4890 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3610918107 0 0) 0x7f70ec06e7c0 con 0x7f70ec0b4890 2024-03-29T11:39:27.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.347+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 >> v1:172.21.15.204:6789/0 conn(0x7f70ec152120 legacy=0x7f70ec0b3b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:27.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 >> v1:172.21.15.22:6789/0 conn(0x7f70ec071a60 legacy=0x7f70ec15a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec06e450 con 0x7f70ec0b4890 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f70dc0019b0 con 0x7f70ec0b4890 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.349+0000 7f70e9ffb640 10 monclient: dump: 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:27.352 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:27.353 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.353 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:27.353 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:27.353 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:27.353 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:27.353 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:27.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.350+0000 7f70f18b3640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:39:27.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.350+0000 7f70f18b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:27.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.350+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70dc001b90 con 0x7f70ec0b4890 2024-03-29T11:39:27.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.350+0000 7f70e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:27.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.350+0000 7f70f18b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:27.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.350+0000 7f70f18b3640 4 set_mon_vals no callback set 2024-03-29T11:39:27.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.351+0000 7f70f18b3640 10 monclient: shutdown 2024-03-29T11:39:27.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.351+0000 7f70f18b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:27.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.351+0000 7f70f18b3640 1 -- 172.21.15.22:0/1164227279 >> v1:172.21.15.22:6790/0 conn(0x7f70ec0b4890 legacy=0x7f70ec071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.351+0000 7f70e9ffb640 1 -- 172.21.15.22:0/1164227279 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f70dc001f60 con 0x7f70ec0b4890 2024-03-29T11:39:27.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.351+0000 7f70e9ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:39:27.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.351+0000 7f70f18b3640 1 -- 172.21.15.22:0/1164227279 shutdown_connections 2024-03-29T11:39:27.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.351+0000 7f70f18b3640 1 -- 172.21.15.22:0/1164227279 >> 172.21.15.22:0/1164227279 conn(0x7f70ec066a00 msgr2=0x7f70ec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:27.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.351+0000 7f70f18b3640 1 -- 172.21.15.22:0/1164227279 shutdown_connections 2024-03-29T11:39:27.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.352+0000 7f70f18b3640 1 -- 172.21.15.22:0/1164227279 wait complete. 2024-03-29T11:39:27.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.352+0000 7f70f18b3640 10 monclient: build_initial_monmap 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.352+0000 7f70f18b3640 10 monclient: monmap: 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.353+0000 7f70f18b3640 1 librados: starting msgr at 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.353+0000 7f70f18b3640 1 librados: starting objecter 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.353+0000 7f70f18b3640 1 Processor -- start 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.353+0000 7f70f18b3640 1 -- start start 2024-03-29T11:39:27.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.353+0000 7f70f18b3640 1 librados: setting wanted keys 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.353+0000 7f70f18b3640 1 librados: calling monclient init 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.353+0000 7f70f18b3640 10 monclient: init 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 10 monclient(hunting): picked mon.noname-b con 0x7f70ec071a60 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 10 monclient(hunting): picked mon.noname-c con 0x7f70ec0b4890 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 10 monclient(hunting): picked mon.noname-a con 0x7f70ec152120 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0b0490 con 0x7f70ec152120 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0acd80 con 0x7f70ec071a60 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0b0700 con 0x7f70ec0b4890 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:27.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.354+0000 7f70f18b3640 10 monclient(hunting): authenticate will time out at 1217.631556s 2024-03-29T11:39:27.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.357+0000 7f70ea7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f70ec0b4890 0x7f70ec147ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42274/0 (socket says 172.21.15.22:42274) 2024-03-29T11:39:27.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.357+0000 7f70ea7fc640 1 -- 172.21.15.22:0/2167884067 learned_addr learned my addr 172.21.15.22:0/2167884067 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:27.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.357+0000 7f70d3fff640 10 client.?.objecter ms_handle_connect 0x7f70ec0b4890 2024-03-29T11:39:27.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.357+0000 7f70d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.358+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159625664 0 0) 0x7f70ec0b0700 con 0x7f70ec0b4890 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.358+0000 7f70d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.358+0000 7f70d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.358+0000 7f70d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.358+0000 7f70d3fff640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.358+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d8004eb0 con 0x7f70ec0b4890 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652571066 0 0) 0x7f70d8004eb0 con 0x7f70ec0b4890 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70ec0b0700 con 0x7f70ec0b4890 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f70e00017f0 con 0x7f70ec0b4890 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 monclient(hunting): dump: 2024-03-29T11:39:27.361 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:27.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:27.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 client.?.objecter ms_handle_connect 0x7f70ec152120 2024-03-29T11:39:27.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:27.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 client.?.objecter ms_handle_connect 0x7f70ec071a60 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3919732987 0 0) 0x7f70ec0b0700 con 0x7f70ec0b4890 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 >> v1:172.21.15.204:6789/0 conn(0x7f70ec071a60 legacy=0x7f70ec0b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 >> v1:172.21.15.22:6789/0 conn(0x7f70ec152120 legacy=0x7f70ec1485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 1 monclient: found mon.c 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec08bb70 con 0x7f70ec0b4890 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.359+0000 7f70d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:39:27.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 10 monclient: _renew_subs 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec08be20 con 0x7f70ec0b4890 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 10 monclient: _renew_subs 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec0b1410 con 0x7f70ec0b4890 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70f18b3640 1 librados: init done 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70e0001950 con 0x7f70ec0b4890 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.360+0000 7f70e97fa640 4 set_mon_vals no callback set 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.361+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f70e00049e0 con 0x7f70ec0b4890 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.361+0000 7f70d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.361+0000 7f70d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.361+0000 7f70d3fff640 10 monclient: dump: 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:27.367 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.363+0000 7f70f18b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.363+0000 7f70f18b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.364+0000 7f70f18b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.364+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc005080 con 0x7f70ec0b4890 2024-03-29T11:39:27.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.364+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f70e0004b40 con 0x7f70ec0b4890 2024-03-29T11:39:27.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.366+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f70e0084a40 con 0x7f70ec0b4890 2024-03-29T11:39:27.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.366+0000 7f70d3fff640 10 client.4415.objecter ms_dispatch 0x7f70ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:27.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.366+0000 7f70d3fff640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:27.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.366+0000 7f70d3fff640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:27.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.366+0000 7f70d3fff640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T11:39:27.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.366+0000 7f70eaffd640 10 monclient: get_auth_request con 0x7f70d806db60 auth_method 0 2024-03-29T11:39:27.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.367+0000 7f70d3fff640 10 client.4415.objecter ms_handle_connect 0x7f70d806db60 2024-03-29T11:39:27.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.372+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70e0050320 con 0x7f70ec0b4890 2024-03-29T11:39:27.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.372+0000 7f70d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.372+0000 7f70d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:27.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.701+0000 7f70f18b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.701+0000 7f70f18b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.701+0000 7f70f18b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:27.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.701+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70cc005330 con 0x7f70ec0b4890 2024-03-29T11:39:27.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.703+0000 7f70d3fff640 1 -- 172.21.15.22:0/2167884067 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f70e0052250 con 0x7f70ec0b4890 2024-03-29T11:39:27.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.703+0000 7f70d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.703+0000 7f70d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:27.704 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:39:27.705 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:27.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.706+0000 7f70f18b3640 10 librados: watch_flush enter 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.706+0000 7f70f18b3640 10 librados: watch_flush exit 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.706+0000 7f70f18b3640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.706+0000 7f70f18b3640 10 client.4415.objecter successfully canceled tick 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.706+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 >> v1:172.21.15.204:6816/3848099522 conn(0x7f70d806db60 legacy=0x7f70d8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.706+0000 7f70f18b3640 10 monclient: shutdown 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.706+0000 7f70f18b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:27.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.706+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 >> v1:172.21.15.22:6790/0 conn(0x7f70ec0b4890 legacy=0x7f70ec147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.707+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 shutdown_connections 2024-03-29T11:39:27.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.707+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 >> 172.21.15.22:0/2167884067 conn(0x7f70ec066a00 msgr2=0x7f70ec069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:27.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.708+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 shutdown_connections 2024-03-29T11:39:27.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.708+0000 7f70f18b3640 1 -- 172.21.15.22:0/2167884067 wait complete. 2024-03-29T11:39:27.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:27.708+0000 7f70f18b3640 1 librados: shutdown 2024-03-29T11:39:27.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:34.534 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:34.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.679+0000 7f2b1a35a640 10 monclient: get_monmap_and_config 2024-03-29T11:39:34.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.679+0000 7f2b1a35a640 10 monclient: build_initial_monmap 2024-03-29T11:39:34.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.679+0000 7f2b1a35a640 10 monclient: monmap: 2024-03-29T11:39:34.680 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:34.680 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:34.680 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:39:34.679556+0000 2024-03-29T11:39:34.681 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:39:34.679556+0000 2024-03-29T11:39:34.681 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:34.681 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:34.681 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:34.681 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:39:34.681 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:39:34.681 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.680+0000 7f2b1a35a640 1 Processor -- start 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.680+0000 7f2b1a35a640 1 -- start start 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.680+0000 7f2b1a35a640 10 monclient: init 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:34.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 10 monclient(hunting): picked mon.noname-c con 0x7f2b0c006e30 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 10 monclient(hunting): picked mon.noname-b con 0x7f2b0c035e00 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 10 monclient(hunting): picked mon.noname-a con 0x7f2b0c0c9670 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c035970 con 0x7f2b0c0c9670 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c036280 con 0x7f2b0c006e30 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c035640 con 0x7f2b0c035e00 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:34.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.681+0000 7f2b1a35a640 10 monclient(hunting): authenticate will time out at 1224.957322s 2024-03-29T11:39:34.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.684+0000 7f2b18b57640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2b0c035e00 0x7f2b0c0c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42670/0 (socket says 172.21.15.22:42670) 2024-03-29T11:39:34.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.684+0000 7f2b18b57640 1 -- 172.21.15.22:0/2154155490 learned_addr learned my addr 172.21.15.22:0/2154155490 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.684+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315409259 0 0) 0x7f2b0c035640 con 0x7f2b0c035e00 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.684+0000 7f2b07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.684+0000 7f2b07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.684+0000 7f2b07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.684+0000 7f2b07fff640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:39:34.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.684+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b00003520 con 0x7f2b0c035e00 2024-03-29T11:39:34.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1338566347 0 0) 0x7f2b00003520 con 0x7f2b0c035e00 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 >> v1:172.21.15.204:6789/0 conn(0x7f2b0c006e30 legacy=0x7f2b0c0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 >> v1:172.21.15.22:6789/0 conn(0x7f2b0c0c9670 legacy=0x7f2b0c0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 1 monclient: found mon.noname-b 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0c034e20 con 0x7f2b0c035e00 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:34.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2b10001690 con 0x7f2b0c035e00 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 10 monclient: dump: 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.688 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b1a35a640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b1a35a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b100018f0 con 0x7f2b0c035e00 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b1a35a640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:34.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.685+0000 7f2b1a35a640 4 set_mon_vals no callback set 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.686+0000 7f2b07fff640 1 -- 172.21.15.22:0/2154155490 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2b10001d00 con 0x7f2b0c035e00 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.686+0000 7f2b07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.686+0000 7f2b07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.686+0000 7f2b07fff640 10 monclient: dump: 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:34.690 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:34.691 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:34.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.686+0000 7f2b1a35a640 10 monclient: shutdown 2024-03-29T11:39:34.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.686+0000 7f2b1a35a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:34.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.686+0000 7f2b1a35a640 1 -- 172.21.15.22:0/2154155490 >> v1:172.21.15.22:6790/0 conn(0x7f2b0c035e00 legacy=0x7f2b0c0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.688+0000 7f2b1a35a640 1 -- 172.21.15.22:0/2154155490 shutdown_connections 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.688+0000 7f2b1a35a640 1 -- 172.21.15.22:0/2154155490 >> 172.21.15.22:0/2154155490 conn(0x7f2b0c005c00 msgr2=0x7f2b0c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.689+0000 7f2b1a35a640 1 -- 172.21.15.22:0/2154155490 shutdown_connections 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.689+0000 7f2b1a35a640 1 -- 172.21.15.22:0/2154155490 wait complete. 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.690+0000 7f2b1a35a640 10 monclient: build_initial_monmap 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.690+0000 7f2b1a35a640 10 monclient: monmap: 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:39:34.698 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.690+0000 7f2b1a35a640 1 librados: starting msgr at 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.690+0000 7f2b1a35a640 1 librados: starting objecter 2024-03-29T11:39:34.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.690+0000 7f2b1a35a640 1 Processor -- start 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 1 -- start start 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 1 librados: setting wanted keys 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 1 librados: calling monclient init 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 10 monclient: init 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:34.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 10 monclient(hunting): picked mon.noname-c con 0x7f2b0c006e30 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 10 monclient(hunting): picked mon.noname-a con 0x7f2b0c035e00 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 10 monclient(hunting): picked mon.noname-b con 0x7f2b0c0c9670 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c0ce820 con 0x7f2b0c035e00 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c0cdf60 con 0x7f2b0c0c9670 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0c0c5430 con 0x7f2b0c006e30 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.691+0000 7f2b1a35a640 10 monclient(hunting): authenticate will time out at 1224.967916s 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.692+0000 7f2b19358640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2b0c006e30 0x7f2b0c0cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:46852/0 (socket says 172.21.15.22:46852) 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.692+0000 7f2b19358640 1 -- 172.21.15.22:0/1124414839 learned_addr learned my addr 172.21.15.22:0/1124414839 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:34.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.693+0000 7f2b05ffb640 10 client.?.objecter ms_handle_connect 0x7f2b0c006e30 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.693+0000 7f2b05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.694+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688387369 0 0) 0x7f2b0c0c5430 con 0x7f2b0c006e30 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.694+0000 7f2b05ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.694+0000 7f2b05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.694+0000 7f2b05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.694+0000 7f2b05ffb640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.694+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af8004eb0 con 0x7f2b0c006e30 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.694+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1415603886 0 0) 0x7f2af8004eb0 con 0x7f2b0c006e30 2024-03-29T11:39:34.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b0c0c5430 con 0x7f2b0c006e30 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2b1404fdd0 con 0x7f2b0c006e30 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:34.703 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3957659065 0 0) 0x7f2b0c0c5430 con 0x7f2b0c006e30 2024-03-29T11:39:34.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 >> v1:172.21.15.204:6789/0 conn(0x7f2b0c0c9670 legacy=0x7f2b0c0c4d20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:39:34.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.695+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 >> v1:172.21.15.22:6789/0 conn(0x7f2b0c035e00 legacy=0x7f2b0c0c4610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b05ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b05ffb640 1 monclient: found mon.c 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b05ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0c0cf3c0 con 0x7f2b0c006e30 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b05ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:34.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b05ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b1a35a640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b1a35a640 10 monclient: _renew_subs 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b1a35a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.698+0000 7f2b1a35a640 1 -- 172.21.15.22:0/1124414839 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0c0ce230 con 0x7f2b0c006e30 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.699+0000 7f2b1a35a640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.699+0000 7f2b1a35a640 10 monclient: _renew_subs 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.699+0000 7f2b1a35a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:34.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.699+0000 7f2b1a35a640 1 -- 172.21.15.22:0/1124414839 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0c0cfb10 con 0x7f2b0c006e30 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.699+0000 7f2b1a35a640 1 librados: init done 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.701+0000 7f2aff7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.701+0000 7f2aff7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.701+0000 7f2aff7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.701+0000 7f2aff7fe640 1 -- 172.21.15.22:0/1124414839 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af4005080 con 0x7f2b0c006e30 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.702+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b1404f400 con 0x7f2b0c006e30 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.702+0000 7f2b05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.702+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2b14075f10 con 0x7f2b0c006e30 2024-03-29T11:39:34.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.702+0000 7f2b05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.702+0000 7f2b05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b05ffb640 10 monclient: dump: 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:34.711 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f2b14067bf0 con 0x7f2b0c006e30 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b077fe640 4 set_mon_vals no callback set 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f2b140ff670 con 0x7f2b0c006e30 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b05ffb640 10 client.4421.objecter ms_dispatch 0x7f2b14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b05ffb640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b05ffb640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:34.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b18b57640 10 monclient: get_auth_request con 0x7f2af806dc60 auth_method 0 2024-03-29T11:39:34.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.703+0000 7f2b05ffb640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:39:34.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.704+0000 7f2b05ffb640 10 client.4421.objecter ms_handle_connect 0x7f2af806dc60 2024-03-29T11:39:34.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.713+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b140cb030 con 0x7f2b0c006e30 2024-03-29T11:39:34.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.713+0000 7f2b05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:34.713+0000 7f2b05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:35.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.052+0000 7f2aff7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.052+0000 7f2aff7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.052+0000 7f2aff7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:39:35.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.052+0000 7f2aff7fe640 1 -- 172.21.15.22:0/1124414839 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2af4005330 con 0x7f2b0c006e30 2024-03-29T11:39:35.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.053+0000 7f2b05ffb640 1 -- 172.21.15.22:0/1124414839 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f2b140ccf60 con 0x7f2b0c006e30 2024-03-29T11:39:35.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.053+0000 7f2b05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.053+0000 7f2b05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:35.054 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:39:35.054 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:35.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.056+0000 7f2aff7fe640 10 librados: watch_flush enter 2024-03-29T11:39:35.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.056+0000 7f2aff7fe640 10 librados: watch_flush exit 2024-03-29T11:39:35.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.056+0000 7f2aff7fe640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:39:35.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.056+0000 7f2aff7fe640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:39:35.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.057+0000 7f2aff7fe640 1 -- 172.21.15.22:0/1124414839 >> v1:172.21.15.204:6816/3848099522 conn(0x7f2af806dc60 legacy=0x7f2af8070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.057+0000 7f2aff7fe640 10 monclient: shutdown 2024-03-29T11:39:35.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.057+0000 7f2aff7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:35.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.057+0000 7f2aff7fe640 1 -- 172.21.15.22:0/1124414839 >> v1:172.21.15.22:6790/0 conn(0x7f2b0c006e30 legacy=0x7f2b0c0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.057+0000 7f2aff7fe640 1 -- 172.21.15.22:0/1124414839 shutdown_connections 2024-03-29T11:39:35.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.058+0000 7f2aff7fe640 1 -- 172.21.15.22:0/1124414839 >> 172.21.15.22:0/1124414839 conn(0x7f2b0c005c00 msgr2=0x7f2b0c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:35.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.058+0000 7f2aff7fe640 1 -- 172.21.15.22:0/1124414839 shutdown_connections 2024-03-29T11:39:35.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.058+0000 7f2aff7fe640 1 -- 172.21.15.22:0/1124414839 wait complete. 2024-03-29T11:39:35.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:35.058+0000 7f2aff7fe640 1 librados: shutdown 2024-03-29T11:39:35.077 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:41.883 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:42.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.036+0000 7fa63eaac640 10 monclient: get_monmap_and_config 2024-03-29T11:39:42.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.036+0000 7fa63eaac640 10 monclient: build_initial_monmap 2024-03-29T11:39:42.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.036+0000 7fa63eaac640 10 monclient: monmap: 2024-03-29T11:39:42.037 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:42.037 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:42.038 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:39:42.037092+0000 2024-03-29T11:39:42.038 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:39:42.037092+0000 2024-03-29T11:39:42.038 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:42.038 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:42.038 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:42.038 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:39:42.038 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:39:42.038 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:42.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.038+0000 7fa63eaac640 1 Processor -- start 2024-03-29T11:39:42.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.039+0000 7fa63eaac640 1 -- start start 2024-03-29T11:39:42.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.039+0000 7fa63eaac640 10 monclient: init 2024-03-29T11:39:42.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:42.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:42.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 10 monclient(hunting): picked mon.noname-a con 0x7fa630006d80 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:42.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 10 monclient(hunting): picked mon.noname-c con 0x7fa6300cc310 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:42.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 10 monclient(hunting): picked mon.noname-b con 0x7fa6300c9430 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:42.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa630008470 con 0x7fa630006d80 2024-03-29T11:39:42.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63002be30 con 0x7fa6300cc310 2024-03-29T11:39:42.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63002c110 con 0x7fa6300c9430 2024-03-29T11:39:42.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:42.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.040+0000 7fa63eaac640 10 monclient(hunting): authenticate will time out at 1232.316407s 2024-03-29T11:39:42.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.043+0000 7fa63d2a9640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa6300cc310 0x7fa6300c8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:33231/0 (socket says 172.21.15.22:33231) 2024-03-29T11:39:42.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.043+0000 7fa63d2a9640 1 -- 172.21.15.22:0/3357053346 learned_addr learned my addr 172.21.15.22:0/3357053346 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233954855 0 0) 0x7fa63002c110 con 0x7fa6300c9430 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa624003520 con 0x7fa6300c9430 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683660006 0 0) 0x7fa63002be30 con 0x7fa6300cc310 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa63002c110 con 0x7fa6300cc310 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3715779194 0 0) 0x7fa624003520 con 0x7fa6300c9430 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 >> v1:172.21.15.204:6789/0 conn(0x7fa6300cc310 legacy=0x7fa6300c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.046+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 >> v1:172.21.15.22:6789/0 conn(0x7fa630006d80 legacy=0x7fa6300cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: _finish_hunting 0 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 1 monclient: found mon.noname-b 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63002c3f0 con 0x7fa6300c9430 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: _finish_auth 0 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: _check_auth_tickets 2024-03-29T11:39:42.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63eaac640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63eaac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa6340016e0 con 0x7fa6300c9430 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: dump: 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:42.052 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63eaac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa634001900 con 0x7fa6300c9430 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 1 -- 172.21.15.22:0/3357053346 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa634001c90 con 0x7fa6300c9430 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63caa8640 10 monclient: dump: 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:42.053 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63eaac640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.047+0000 7fa63eaac640 4 set_mon_vals no callback set 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.048+0000 7fa63eaac640 10 monclient: shutdown 2024-03-29T11:39:42.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.048+0000 7fa63eaac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.048+0000 7fa63eaac640 1 -- 172.21.15.22:0/3357053346 >> v1:172.21.15.22:6790/0 conn(0x7fa6300c9430 legacy=0x7fa6300d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.048+0000 7fa63eaac640 1 -- 172.21.15.22:0/3357053346 shutdown_connections 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.048+0000 7fa63eaac640 1 -- 172.21.15.22:0/3357053346 >> 172.21.15.22:0/3357053346 conn(0x7fa630005c00 msgr2=0x7fa630006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:42.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.049+0000 7fa63eaac640 1 -- 172.21.15.22:0/3357053346 shutdown_connections 2024-03-29T11:39:42.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.049+0000 7fa63eaac640 1 -- 172.21.15.22:0/3357053346 wait complete. 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 10 monclient: build_initial_monmap 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 10 monclient: monmap: 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:42.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 1 librados: starting msgr at 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 1 librados: starting objecter 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 1 Processor -- start 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 1 -- start start 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 1 librados: setting wanted keys 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 1 librados: calling monclient init 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.050+0000 7fa63eaac640 10 monclient: init 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.051+0000 7fa63eaac640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.051+0000 7fa63eaac640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.051+0000 7fa63eaac640 10 monclient(hunting): picked mon.noname-c con 0x7fa630006d80 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.051+0000 7fa63eaac640 10 monclient(hunting): picked mon.noname-b con 0x7fa6300c9430 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:42.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.051+0000 7fa63eaac640 10 monclient(hunting): picked mon.noname-a con 0x7fa6300cc310 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.055+0000 7fa63daaa640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa630006d80 0x7fa6300a64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:43207/0 (socket says 172.21.15.22:43207) 2024-03-29T11:39:42.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.055+0000 7fa63daaa640 1 -- 172.21.15.22:0/1854198370 learned_addr learned my addr 172.21.15.22:0/1854198370 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:42.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.058+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300a6fd0 con 0x7fa6300cc310 2024-03-29T11:39:42.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.058+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300a7d80 con 0x7fa6300c9430 2024-03-29T11:39:42.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.058+0000 7fa62e7fc640 10 client.?.objecter ms_handle_connect 0x7fa630006d80 2024-03-29T11:39:42.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.059+0000 7fa62e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.059+0000 7fa62e7fc640 10 client.?.objecter ms_handle_connect 0x7fa6300c9430 2024-03-29T11:39:42.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.059+0000 7fa62e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.059+0000 7fa62e7fc640 10 client.?.objecter ms_handle_connect 0x7fa6300cc310 2024-03-29T11:39:42.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.059+0000 7fa62e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6300a8060 con 0x7fa630006d80 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa63eaac640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa63eaac640 10 monclient(hunting): authenticate will time out at 1232.337687s 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423980790 0 0) 0x7fa6300a7d80 con 0x7fa6300c9430 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa61c004eb0 con 0x7fa6300c9430 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321342710 0 0) 0x7fa6300a6fd0 con 0x7fa6300cc310 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:39:42.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.061+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6300a7d80 con 0x7fa6300cc310 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4091899153 0 0) 0x7fa61c004eb0 con 0x7fa6300c9430 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6300a6fd0 con 0x7fa6300c9430 2024-03-29T11:39:42.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821937969 0 0) 0x7fa6300a7d80 con 0x7fa6300cc310 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa61c004eb0 con 0x7fa6300cc310 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa628001930 con 0x7fa6300c9430 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): dump: 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:42.069 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa6340015a0 con 0x7fa6300cc310 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): dump: 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:42.070 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019447633 0 0) 0x7fa6300a8060 con 0x7fa630006d80 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6300a7d80 con 0x7fa630006d80 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2736203441 0 0) 0x7fa61c004eb0 con 0x7fa6300cc310 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 >> v1:172.21.15.22:6790/0 conn(0x7fa630006d80 legacy=0x7fa6300a64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 >> v1:172.21.15.204:6789/0 conn(0x7fa6300c9430 legacy=0x7fa6300bdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:42.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 monclient: found mon.a 2024-03-29T11:39:42.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:42.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:42.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6300a6bd0 con 0x7fa6300cc310 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa634001700 con 0x7fa6300cc310 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.064+0000 7fa62f7fe640 4 set_mon_vals no callback set 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 10 monclient: _renew_subs 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6300a5a00 con 0x7fa6300cc310 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 10 monclient: _renew_subs 2024-03-29T11:39:42.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6300be660 con 0x7fa6300cc310 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.065+0000 7fa63eaac640 1 librados: init done 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.066+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa634003b50 con 0x7fa6300cc310 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.066+0000 7fa62e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.066+0000 7fa62e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.066+0000 7fa62e7fc640 10 monclient: dump: 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:42.076 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.068+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fa634019b20 con 0x7fa6300cc310 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fa634053640 con 0x7fa6300cc310 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa62e7fc640 10 client.4410.objecter ms_dispatch 0x7fa638000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa62e7fc640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa62e7fc640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa62e7fc640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa63daaa640 10 monclient: get_auth_request con 0x7fa61c06ded0 auth_method 0 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa63eaac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa63eaac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa63eaac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa62e7fc640 10 client.4410.objecter ms_handle_connect 0x7fa61c06ded0 2024-03-29T11:39:42.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.069+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6300be960 con 0x7fa6300cc310 2024-03-29T11:39:42.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.078+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa63404f8e0 con 0x7fa6300cc310 2024-03-29T11:39:42.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.078+0000 7fa62e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.078+0000 7fa62e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:42.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.414+0000 7fa63eaac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.414+0000 7fa63eaac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.414+0000 7fa63eaac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:42.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.414+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6300a6d60 con 0x7fa6300cc310 2024-03-29T11:39:42.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.417+0000 7fa62e7fc640 1 -- 172.21.15.22:0/1854198370 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fa63404fa80 con 0x7fa6300cc310 2024-03-29T11:39:42.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.417+0000 7fa62e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.417+0000 7fa62e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:42.418 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:39:42.418 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:42.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.421+0000 7fa63eaac640 10 librados: watch_flush enter 2024-03-29T11:39:42.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.421+0000 7fa63eaac640 10 librados: watch_flush exit 2024-03-29T11:39:42.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.421+0000 7fa63eaac640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T11:39:42.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.421+0000 7fa63eaac640 10 client.4410.objecter successfully canceled tick 2024-03-29T11:39:42.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.421+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 >> v1:172.21.15.204:6816/3848099522 conn(0x7fa61c06ded0 legacy=0x7fa61c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.421+0000 7fa63eaac640 10 monclient: shutdown 2024-03-29T11:39:42.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.421+0000 7fa63eaac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:42.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.421+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 >> v1:172.21.15.22:6789/0 conn(0x7fa6300cc310 legacy=0x7fa630185420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.425+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 shutdown_connections 2024-03-29T11:39:42.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.425+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 >> 172.21.15.22:0/1854198370 conn(0x7fa630005c00 msgr2=0x7fa6300c8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:42.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.425+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 shutdown_connections 2024-03-29T11:39:42.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.425+0000 7fa63eaac640 1 -- 172.21.15.22:0/1854198370 wait complete. 2024-03-29T11:39:42.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:42.425+0000 7fa63eaac640 1 librados: shutdown 2024-03-29T11:39:42.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:49.250 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:49.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.405+0000 7f97d5b3b640 10 monclient: get_monmap_and_config 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.405+0000 7f97d5b3b640 10 monclient: build_initial_monmap 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.405+0000 7f97d5b3b640 10 monclient: monmap: 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:39:49.405601+0000 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:39:49.405601+0000 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:49.406 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:39:49.407 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:39:49.407 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:49.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.406+0000 7f97d5b3b640 1 Processor -- start 2024-03-29T11:39:49.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.406+0000 7f97d5b3b640 1 -- start start 2024-03-29T11:39:49.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.406+0000 7f97d5b3b640 10 monclient: init 2024-03-29T11:39:49.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 10 monclient(hunting): picked mon.noname-a con 0x7f97d01422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 10 monclient(hunting): picked mon.noname-b con 0x7f97d0142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 10 monclient(hunting): picked mon.noname-c con 0x7f97d00b17d0 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d0142c80 con 0x7f97d01422b0 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d00aae40 con 0x7f97d00b17d0 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d013fbd0 con 0x7f97d0142690 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97d5b3b640 10 monclient(hunting): authenticate will time out at 1239.683746s 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97cf7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f97d01422b0 0x7f97d013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:45452/0 (socket says 172.21.15.22:45452) 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97cf7fe640 1 -- 172.21.15.22:0/796356438 learned_addr learned my addr 172.21.15.22:0/796356438 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.407+0000 7f97ceffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f97d0142690 0x7f97d00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52728/0 (socket says 172.21.15.22:52728) 2024-03-29T11:39:49.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644491278 0 0) 0x7f97d0142c80 con 0x7f97d01422b0 2024-03-29T11:39:49.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:39:49.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b8003520 con 0x7f97d01422b0 2024-03-29T11:39:49.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142520267 0 0) 0x7f97d013fbd0 con 0x7f97d0142690 2024-03-29T11:39:49.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.408+0000 7f97ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:39:49.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d0142c80 con 0x7f97d0142690 2024-03-29T11:39:49.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097932251 0 0) 0x7f97b8003520 con 0x7f97d01422b0 2024-03-29T11:39:49.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 >> v1:172.21.15.22:6790/0 conn(0x7f97d0142690 legacy=0x7f97d00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 >> v1:172.21.15.204:6789/0 conn(0x7f97d00b17d0 legacy=0x7f97d00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 monclient: found mon.noname-a 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d0151940 con 0x7f97d01422b0 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f97c0001690 con 0x7f97d01422b0 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: dump: 2024-03-29T11:39:49.412 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97d5b3b640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97d5b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97c00018b0 con 0x7f97d01422b0 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:49.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 1 -- 172.21.15.22:0/796356438 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f97c0001bc0 con 0x7f97d01422b0 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97ce7fc640 10 monclient: dump: 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:49.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97d5b3b640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:49.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.409+0000 7f97d5b3b640 4 set_mon_vals no callback set 2024-03-29T11:39:49.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.410+0000 7f97d5b3b640 10 monclient: shutdown 2024-03-29T11:39:49.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.410+0000 7f97d5b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:49.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.410+0000 7f97d5b3b640 1 -- 172.21.15.22:0/796356438 >> v1:172.21.15.22:6789/0 conn(0x7f97d01422b0 legacy=0x7f97d013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.410+0000 7f97d5b3b640 1 -- 172.21.15.22:0/796356438 shutdown_connections 2024-03-29T11:39:49.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.410+0000 7f97d5b3b640 1 -- 172.21.15.22:0/796356438 >> 172.21.15.22:0/796356438 conn(0x7f97d0066a00 msgr2=0x7f97d0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:49.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.410+0000 7f97d5b3b640 1 -- 172.21.15.22:0/796356438 shutdown_connections 2024-03-29T11:39:49.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.410+0000 7f97d5b3b640 1 -- 172.21.15.22:0/796356438 wait complete. 2024-03-29T11:39:49.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.411+0000 7f97d5b3b640 10 monclient: build_initial_monmap 2024-03-29T11:39:49.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.411+0000 7f97d5b3b640 10 monclient: monmap: 2024-03-29T11:39:49.421 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:49.421 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.411+0000 7f97d5b3b640 1 librados: starting msgr at 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.412+0000 7f97d5b3b640 1 librados: starting objecter 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.412+0000 7f97d5b3b640 1 Processor -- start 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.412+0000 7f97d5b3b640 1 -- start start 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.412+0000 7f97d5b3b640 1 librados: setting wanted keys 2024-03-29T11:39:49.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.412+0000 7f97d5b3b640 1 librados: calling monclient init 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.412+0000 7f97d5b3b640 10 monclient: init 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 10 monclient(hunting): picked mon.noname-a con 0x7f97d00b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 10 monclient(hunting): picked mon.noname-c con 0x7f97d01422b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 10 monclient(hunting): picked mon.noname-b con 0x7f97d0142690 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d00b1090 con 0x7f97d00b17d0 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d00b1370 con 0x7f97d0142690 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d00b01f0 con 0x7f97d01422b0 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97d5b3b640 10 monclient(hunting): authenticate will time out at 1239.689360s 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97cf7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f97d00b17d0 0x7f97d00afa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:47532/0 (socket says 172.21.15.22:47532) 2024-03-29T11:39:49.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97cf7fe640 1 -- 172.21.15.22:0/3875819788 learned_addr learned my addr 172.21.15.22:0/3875819788 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97b7fff640 10 client.?.objecter ms_handle_connect 0x7f97d00b17d0 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.413+0000 7f97b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 client.?.objecter ms_handle_connect 0x7f97d01422b0 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 client.?.objecter ms_handle_connect 0x7f97d0142690 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216880056 0 0) 0x7f97d00b1090 con 0x7f97d00b17d0 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97bc004eb0 con 0x7f97d00b17d0 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027617185 0 0) 0x7f97d00b01f0 con 0x7f97d01422b0 2024-03-29T11:39:49.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.414+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d00b1090 con 0x7f97d01422b0 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1057757292 0 0) 0x7f97bc004eb0 con 0x7f97d00b17d0 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97d00b01f0 con 0x7f97d00b17d0 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f97c0001610 con 0x7f97d00b17d0 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 10 monclient(hunting): dump: 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:49.425 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578656958 0 0) 0x7f97d00b1370 con 0x7f97d0142690 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97bc004eb0 con 0x7f97d0142690 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2374731010 0 0) 0x7f97d00b1090 con 0x7f97d01422b0 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97d00b1370 con 0x7f97d01422b0 2024-03-29T11:39:49.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3648074989 0 0) 0x7f97d00b01f0 con 0x7f97d00b17d0 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.415+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 >> v1:172.21.15.22:6790/0 conn(0x7f97d01422b0 legacy=0x7f97d013bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 >> v1:172.21.15.204:6789/0 conn(0x7f97d0142690 legacy=0x7f97d01dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 1 monclient: found mon.a 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d00b0380 con 0x7f97d00b17d0 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 10 monclient: _renew_subs 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d00aef80 con 0x7f97d00b17d0 2024-03-29T11:39:49.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 10 monclient: _renew_subs 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d013c630 con 0x7f97d00b17d0 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97d5b3b640 1 librados: init done 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.416+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97c0004a90 con 0x7f97d00b17d0 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.417+0000 7f97b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.417+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f97c0004de0 con 0x7f97d00b17d0 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.417+0000 7f97b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.417+0000 7f97b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.417+0000 7f97cdffb640 4 set_mon_vals no callback set 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.417+0000 7f97b7fff640 10 monclient: dump: 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:49.428 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.418+0000 7f97d5b3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.418+0000 7f97d5b3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.418+0000 7f97d5b3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.418+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d0075f10 con 0x7f97d00b17d0 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.419+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f97c0004f40 con 0x7f97d00b17d0 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.420+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f97c0084b40 con 0x7f97d00b17d0 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.420+0000 7f97b7fff640 10 client.4416.objecter ms_dispatch 0x7f97d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.420+0000 7f97b7fff640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.420+0000 7f97b7fff640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:49.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.420+0000 7f97b7fff640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:39:49.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.420+0000 7f97ceffd640 10 monclient: get_auth_request con 0x7f97bc06e030 auth_method 0 2024-03-29T11:39:49.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.421+0000 7f97b7fff640 10 client.4416.objecter ms_handle_connect 0x7f97bc06e030 2024-03-29T11:39:49.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.427+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97c0050420 con 0x7f97d00b17d0 2024-03-29T11:39:49.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.427+0000 7f97b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.427+0000 7f97b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:49.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.775+0000 7f97d5b3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:49.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.775+0000 7f97d5b3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:49.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.775+0000 7f97d5b3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:49.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.775+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97d00b2270 con 0x7f97d00b17d0 2024-03-29T11:39:49.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.776+0000 7f97b7fff640 1 -- 172.21.15.22:0/3875819788 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f97c0052350 con 0x7f97d00b17d0 2024-03-29T11:39:49.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.776+0000 7f97b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:49.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.776+0000 7f97b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:49.777 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:39:49.777 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.779+0000 7f97d5b3b640 10 librados: watch_flush enter 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.779+0000 7f97d5b3b640 10 librados: watch_flush exit 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.779+0000 7f97d5b3b640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.779+0000 7f97d5b3b640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 >> v1:172.21.15.204:6816/3848099522 conn(0x7f97bc06e030 legacy=0x7f97bc0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 10 monclient: shutdown 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 >> v1:172.21.15.22:6789/0 conn(0x7f97d00b17d0 legacy=0x7f97d00afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 shutdown_connections 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 >> 172.21.15.22:0/3875819788 conn(0x7f97d0066a00 msgr2=0x7f97d013d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 shutdown_connections 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 1 -- 172.21.15.22:0/3875819788 wait complete. 2024-03-29T11:39:49.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:49.780+0000 7f97d5b3b640 1 librados: shutdown 2024-03-29T11:39:49.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:56.603 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:56.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.750+0000 7f307fd8c640 10 monclient: get_monmap_and_config 2024-03-29T11:39:56.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.750+0000 7f307fd8c640 10 monclient: build_initial_monmap 2024-03-29T11:39:56.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.750+0000 7f307fd8c640 10 monclient: monmap: 2024-03-29T11:39:56.752 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:56.752 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:56.752 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:39:56.751781+0000 2024-03-29T11:39:56.753 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:39:56.751781+0000 2024-03-29T11:39:56.753 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:56.753 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:56.753 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:56.753 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:39:56.753 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:39:56.753 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:56.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.753+0000 7f307fd8c640 1 Processor -- start 2024-03-29T11:39:56.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.753+0000 7f307fd8c640 1 -- start start 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.753+0000 7f307fd8c640 10 monclient: init 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 10 monclient(hunting): picked mon.noname-c con 0x7f30780b0770 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 10 monclient(hunting): picked mon.noname-b con 0x7f307814a700 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 10 monclient(hunting): picked mon.noname-a con 0x7f30780ac5a0 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30780aba70 con 0x7f30780ac5a0 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30780aecb0 con 0x7f30780b0770 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30780af150 con 0x7f307814a700 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:56.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.754+0000 7f307fd8c640 10 monclient(hunting): authenticate will time out at 1247.031525s 2024-03-29T11:39:56.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.758+0000 7f307d327640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f307814a700 0x7f30780abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:44326/0 (socket says 172.21.15.22:44326) 2024-03-29T11:39:56.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.758+0000 7f307d327640 1 -- 172.21.15.22:0/1295150526 learned_addr learned my addr 172.21.15.22:0/1295150526 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:56.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.758+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997030477 0 0) 0x7f30780af150 con 0x7f307814a700 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.758+0000 7f307cb26640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.758+0000 7f307cb26640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.758+0000 7f307cb26640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.758+0000 7f307cb26640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3064003520 con 0x7f307814a700 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558122292 0 0) 0x7f30780aba70 con 0x7f30780ac5a0 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:39:56.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30780af150 con 0x7f30780ac5a0 2024-03-29T11:39:56.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659780961 0 0) 0x7f3064003520 con 0x7f307814a700 2024-03-29T11:39:56.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.759+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 >> v1:172.21.15.204:6789/0 conn(0x7f30780b0770 legacy=0x7f3078149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:56.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 >> v1:172.21.15.22:6789/0 conn(0x7f30780ac5a0 legacy=0x7f3078151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:56.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 10 monclient: _finish_hunting 0 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 1 monclient: found mon.noname-b 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30780ac9d0 con 0x7f307814a700 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 10 monclient: _finish_auth 0 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 10 monclient: _check_auth_tickets 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3068001690 con 0x7f307814a700 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:56.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.761+0000 7f307cb26640 10 monclient: dump: 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307fd8c640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:39:56.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307fd8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:56.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30680017f0 con 0x7f307814a700 2024-03-29T11:39:56.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307cb26640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:56.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307cb26640 1 -- 172.21.15.22:0/1295150526 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3068001c40 con 0x7f307814a700 2024-03-29T11:39:56.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307cb26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307cb26640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:56.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307cb26640 10 monclient: dump: 2024-03-29T11:39:56.768 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307fd8c640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.763+0000 7f307fd8c640 4 set_mon_vals no callback set 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.764+0000 7f307fd8c640 10 monclient: shutdown 2024-03-29T11:39:56.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.764+0000 7f307fd8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.764+0000 7f307fd8c640 1 -- 172.21.15.22:0/1295150526 >> v1:172.21.15.22:6790/0 conn(0x7f307814a700 legacy=0x7f30780abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.764+0000 7f307fd8c640 1 -- 172.21.15.22:0/1295150526 shutdown_connections 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.764+0000 7f307fd8c640 1 -- 172.21.15.22:0/1295150526 >> 172.21.15.22:0/1295150526 conn(0x7f307806b660 msgr2=0x7f307806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.765+0000 7f307fd8c640 1 -- 172.21.15.22:0/1295150526 shutdown_connections 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.765+0000 7f307fd8c640 1 -- 172.21.15.22:0/1295150526 wait complete. 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.765+0000 7f307fd8c640 10 monclient: build_initial_monmap 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.765+0000 7f307fd8c640 10 monclient: monmap: 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:56.770 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.766+0000 7f307fd8c640 1 librados: starting msgr at 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.766+0000 7f307fd8c640 1 librados: starting objecter 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.766+0000 7f307fd8c640 1 Processor -- start 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.766+0000 7f307fd8c640 1 -- start start 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.766+0000 7f307fd8c640 1 librados: setting wanted keys 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.766+0000 7f307fd8c640 1 librados: calling monclient init 2024-03-29T11:39:56.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.766+0000 7f307fd8c640 10 monclient: init 2024-03-29T11:39:56.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:56.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:56.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 10 monclient(hunting): picked mon.noname-c con 0x7f30780ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:39:56.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 10 monclient(hunting): picked mon.noname-a con 0x7f30780b0770 addr v1:172.21.15.22:6789/0 2024-03-29T11:39:56.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 10 monclient(hunting): picked mon.noname-b con 0x7f307814a700 addr v1:172.21.15.204:6789/0 2024-03-29T11:39:56.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3078127fc0 con 0x7f30780b0770 2024-03-29T11:39:56.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3078125c40 con 0x7f307814a700 2024-03-29T11:39:56.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30781239d0 con 0x7f30780ac5a0 2024-03-29T11:39:56.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:56.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307fd8c640 10 monclient(hunting): authenticate will time out at 1247.044377s 2024-03-29T11:39:56.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307db28640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f30780ac5a0 0x7f30781dba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47752/0 (socket says 172.21.15.22:47752) 2024-03-29T11:39:56.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.767+0000 7f307db28640 1 -- 172.21.15.22:0/333325815 learned_addr learned my addr 172.21.15.22:0/333325815 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:39:56.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 client.?.objecter ms_handle_connect 0x7f30780ac5a0 2024-03-29T11:39:56.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:56.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 client.?.objecter ms_handle_connect 0x7f307814a700 2024-03-29T11:39:56.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:56.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463340889 0 0) 0x7f30781239d0 con 0x7f30780ac5a0 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f305c004eb0 con 0x7f30780ac5a0 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 client.?.objecter ms_handle_connect 0x7f30780b0770 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701247617 0 0) 0x7f3078125c40 con 0x7f307814a700 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.771+0000 7f306e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:39:56.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30781239d0 con 0x7f307814a700 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197109953 0 0) 0x7f3078127fc0 con 0x7f30780b0770 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.772+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3078125c40 con 0x7f30780b0770 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.773+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 667950268 0 0) 0x7f3078125c40 con 0x7f30780b0770 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.773+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3078127fc0 con 0x7f30780b0770 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.773+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3068001b00 con 0x7f30780b0770 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.773+0000 7f306e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.773+0000 7f306e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:56.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.773+0000 7f306e7fc640 10 monclient(hunting): dump: 2024-03-29T11:39:56.781 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:56.781 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:56.781 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:56.781 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:56.781 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.773+0000 7f306e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3867599997 0 0) 0x7f3078127fc0 con 0x7f30780b0770 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 >> v1:172.21.15.22:6790/0 conn(0x7f30780ac5a0 legacy=0x7f30781dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 >> v1:172.21.15.204:6789/0 conn(0x7f307814a700 legacy=0x7f30781e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 1 monclient: found mon.a 2024-03-29T11:39:56.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:56.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:56.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078123b60 con 0x7f30780b0770 2024-03-29T11:39:56.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:56.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:56.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.774+0000 7f306e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:56.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:39:56.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 10 monclient: _renew_subs 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30781dc180 con 0x7f30780b0770 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 10 monclient: _renew_subs 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3078126f80 con 0x7f30780b0770 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.775+0000 7f307fd8c640 1 librados: init done 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.777+0000 7f307fd8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:56.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.777+0000 7f307fd8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.777+0000 7f307fd8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.777+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30780ab560 con 0x7f30780b0770 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.778+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3068001d00 con 0x7f30780b0770 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.778+0000 7f306e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.778+0000 7f306ffff640 4 set_mon_vals no callback set 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.779+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3068004a90 con 0x7f30780b0770 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.779+0000 7f306e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.779+0000 7f306e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.779+0000 7f306e7fc640 10 monclient: dump: 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:39:56.787 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.779+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f306801aae0 con 0x7f30780b0770 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.780+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f3068084920 con 0x7f30780b0770 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.780+0000 7f306e7fc640 10 client.4422.objecter ms_dispatch 0x7f3078000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.780+0000 7f306e7fc640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.780+0000 7f306e7fc640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.780+0000 7f306e7fc640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.780+0000 7f307db28640 10 monclient: get_auth_request con 0x7f305c06dc30 auth_method 0 2024-03-29T11:39:56.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.781+0000 7f306e7fc640 10 client.4422.objecter ms_handle_connect 0x7f305c06dc30 2024-03-29T11:39:56.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.791+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3068050280 con 0x7f30780b0770 2024-03-29T11:39:56.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.791+0000 7f306e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:56.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:56.791+0000 7f306e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:57.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.139+0000 7f307fd8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.139+0000 7f307fd8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.139+0000 7f307fd8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:39:57.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.139+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30780afd20 con 0x7f30780b0770 2024-03-29T11:39:57.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.141+0000 7f306e7fc640 1 -- 172.21.15.22:0/333325815 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f30680521b0 con 0x7f30780b0770 2024-03-29T11:39:57.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.141+0000 7f306e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.141+0000 7f306e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:57.143 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:39:57.143 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:57.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 10 librados: watch_flush enter 2024-03-29T11:39:57.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 10 librados: watch_flush exit 2024-03-29T11:39:57.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-29T11:39:57.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 10 client.4422.objecter successfully canceled tick 2024-03-29T11:39:57.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 >> v1:172.21.15.204:6816/3848099522 conn(0x7f305c06dc30 legacy=0x7f305c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 10 monclient: shutdown 2024-03-29T11:39:57.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:57.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 >> v1:172.21.15.22:6789/0 conn(0x7f30780b0770 legacy=0x7f3078126690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 shutdown_connections 2024-03-29T11:39:57.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 >> 172.21.15.22:0/333325815 conn(0x7f307806b660 msgr2=0x7f30780713f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:57.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 shutdown_connections 2024-03-29T11:39:57.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.144+0000 7f307fd8c640 1 -- 172.21.15.22:0/333325815 wait complete. 2024-03-29T11:39:57.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:39:57.145+0000 7f307fd8c640 1 librados: shutdown 2024-03-29T11:39:57.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:03.968 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:04.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.123+0000 7f08fee31640 10 monclient: get_monmap_and_config 2024-03-29T11:40:04.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.123+0000 7f08fee31640 10 monclient: build_initial_monmap 2024-03-29T11:40:04.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.123+0000 7f08fee31640 10 monclient: monmap: 2024-03-29T11:40:04.125 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:04.125 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:04.125 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:40:04.124804+0000 2024-03-29T11:40:04.126 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:40:04.124804+0000 2024-03-29T11:40:04.126 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:04.126 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:04.126 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:04.126 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:40:04.126 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:40:04.126 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:04.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.125+0000 7f08fee31640 1 Processor -- start 2024-03-29T11:40:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.125+0000 7f08fee31640 1 -- start start 2024-03-29T11:40:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.125+0000 7f08fee31640 10 monclient: init 2024-03-29T11:40:04.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:04.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:04.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 10 monclient(hunting): picked mon.noname-c con 0x7f08f80b4710 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:04.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 10 monclient(hunting): picked mon.noname-b con 0x7f08f80b3970 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 10 monclient(hunting): picked mon.noname-a con 0x7f08f813f390 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80af020 con 0x7f08f813f390 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80b0910 con 0x7f08f80b4710 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80b0bf0 con 0x7f08f80b3970 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.127+0000 7f08fee31640 10 monclient(hunting): authenticate will time out at 1254.404366s 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.128+0000 7f08fd3ce640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f08f813f390 0x7f08f813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55592/0 (socket says 172.21.15.22:55592) 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.128+0000 7f08fd3ce640 1 -- 172.21.15.22:0/3121449232 learned_addr learned my addr 172.21.15.22:0/3121449232 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.129+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229801296 0 0) 0x7f08f80af020 con 0x7f08f813f390 2024-03-29T11:40:04.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.129+0000 7f08f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.129+0000 7f08f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.129+0000 7f08f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.129+0000 7f08f77fe640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:40:04.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.129+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e0003520 con 0x7f08f813f390 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665995999 0 0) 0x7f08f80b0bf0 con 0x7f08f80b3970 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f80af020 con 0x7f08f80b3970 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803116896 0 0) 0x7f08e0003520 con 0x7f08f813f390 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 >> v1:172.21.15.22:6790/0 conn(0x7f08f80b3970 legacy=0x7f08f80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.130+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 >> v1:172.21.15.204:6789/0 conn(0x7f08f80b4710 legacy=0x7f08f80b1010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 1 monclient: found mon.noname-a 2024-03-29T11:40:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f80aeb00 con 0x7f08f813f390 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08fee31640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08fee31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08ec001610 con 0x7f08f813f390 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.132+0000 7f08f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08f77fe640 10 monclient: dump: 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.136 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08fee31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08ec001830 con 0x7f08f813f390 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08fee31640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08fee31640 4 set_mon_vals no callback set 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08f77fe640 1 -- 172.21.15.22:0/3121449232 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08ec001bc0 con 0x7f08f813f390 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.133+0000 7f08f77fe640 10 monclient: dump: 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:04.138 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.134+0000 7f08fee31640 10 monclient: shutdown 2024-03-29T11:40:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.134+0000 7f08fee31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.134+0000 7f08fee31640 1 -- 172.21.15.22:0/3121449232 >> v1:172.21.15.22:6789/0 conn(0x7f08f813f390 legacy=0x7f08f813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.134+0000 7f08fee31640 1 -- 172.21.15.22:0/3121449232 shutdown_connections 2024-03-29T11:40:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.134+0000 7f08fee31640 1 -- 172.21.15.22:0/3121449232 >> 172.21.15.22:0/3121449232 conn(0x7f08f8066a00 msgr2=0x7f08f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.135+0000 7f08fee31640 1 -- 172.21.15.22:0/3121449232 shutdown_connections 2024-03-29T11:40:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.136+0000 7f08fee31640 1 -- 172.21.15.22:0/3121449232 wait complete. 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.137+0000 7f08fee31640 10 monclient: build_initial_monmap 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.137+0000 7f08fee31640 10 monclient: monmap: 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.138+0000 7f08fee31640 1 librados: starting msgr at 2024-03-29T11:40:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.138+0000 7f08fee31640 1 librados: starting objecter 2024-03-29T11:40:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.138+0000 7f08fee31640 1 Processor -- start 2024-03-29T11:40:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.140+0000 7f08fee31640 1 -- start start 2024-03-29T11:40:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.140+0000 7f08fee31640 1 librados: setting wanted keys 2024-03-29T11:40:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.140+0000 7f08fee31640 1 librados: calling monclient init 2024-03-29T11:40:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.140+0000 7f08fee31640 10 monclient: init 2024-03-29T11:40:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 10 monclient(hunting): picked mon.noname-a con 0x7f08f80b3970 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 10 monclient(hunting): picked mon.noname-b con 0x7f08f80b4710 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 10 monclient(hunting): picked mon.noname-c con 0x7f08f813f390 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f8131630 con 0x7f08f80b3970 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f8141380 con 0x7f08f80b4710 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f8141660 con 0x7f08f813f390 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.141+0000 7f08fee31640 10 monclient(hunting): authenticate will time out at 1254.418799s 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.142+0000 7f08fd3ce640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f08f813f390 0x7f08f8130f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60060/0 (socket says 172.21.15.22:60060) 2024-03-29T11:40:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.142+0000 7f08fd3ce640 1 -- 172.21.15.22:0/2468722243 learned_addr learned my addr 172.21.15.22:0/2468722243 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.145+0000 7f08f57fa640 10 client.?.objecter ms_handle_connect 0x7f08f813f390 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.145+0000 7f08f57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.145+0000 7f08f57fa640 10 client.?.objecter ms_handle_connect 0x7f08f80b3970 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028866326 0 0) 0x7f08f8131630 con 0x7f08f80b3970 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:40:04.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e4004eb0 con 0x7f08f80b3970 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800533079 0 0) 0x7f08f8141660 con 0x7f08f813f390 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8131630 con 0x7f08f813f390 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 client.?.objecter ms_handle_connect 0x7f08f80b4710 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.146+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853601467 0 0) 0x7f08f8141380 con 0x7f08f80b4710 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8141660 con 0x7f08f80b4710 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160290988 0 0) 0x7f08f8131630 con 0x7f08f813f390 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08f8141380 con 0x7f08f813f390 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08ec001b40 con 0x7f08f813f390 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): dump: 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:04.151 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1519272686 0 0) 0x7f08f8141660 con 0x7f08f80b4710 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08f8131630 con 0x7f08f80b4710 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08e8001860 con 0x7f08f80b4710 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.147+0000 7f08f57fa640 10 monclient(hunting): dump: 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.152 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:04.153 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:04.153 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:04.153 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:04.153 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3394791516 0 0) 0x7f08f8141380 con 0x7f08f813f390 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 >> v1:172.21.15.204:6789/0 conn(0x7f08f80b4710 legacy=0x7f08f8130810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 >> v1:172.21.15.22:6789/0 conn(0x7f08f80b3970 legacy=0x7f08f8141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 1 monclient: found mon.c 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f8140010 con 0x7f08f813f390 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 10 monclient: _finish_auth 0 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.148+0000 7f08f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:04.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.149+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08ec004a90 con 0x7f08f813f390 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.149+0000 7f08f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.149+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08ec001ca0 con 0x7f08f813f390 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.149+0000 7f08f6ffd640 4 set_mon_vals no callback set 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.150+0000 7f08f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.150+0000 7f08f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.150+0000 7f08f57fa640 10 monclient: dump: 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:04.156 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.150+0000 7f08fee31640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.150+0000 7f08fee31640 10 monclient: _renew_subs 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.150+0000 7f08fee31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.150+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f814e760 con 0x7f08f813f390 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.151+0000 7f08fee31640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.151+0000 7f08fee31640 10 monclient: _renew_subs 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.151+0000 7f08fee31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.151+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f81394d0 con 0x7f08f813f390 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.151+0000 7f08fee31640 1 librados: init done 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.152+0000 7f08fee31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.152+0000 7f08fee31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.152+0000 7f08fee31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:04.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.152+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f80688a0 con 0x7f08f813f390 2024-03-29T11:40:04.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.154+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f08ec01ac10 con 0x7f08f813f390 2024-03-29T11:40:04.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.155+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f08ec0849a0 con 0x7f08f813f390 2024-03-29T11:40:04.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.155+0000 7f08f57fa640 10 client.4445.objecter ms_dispatch 0x7f08f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:04.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.155+0000 7f08f57fa640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:04.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.155+0000 7f08f57fa640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:04.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.155+0000 7f08fcbcd640 10 monclient: get_auth_request con 0x7f08e406df80 auth_method 0 2024-03-29T11:40:04.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.155+0000 7f08f57fa640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:40:04.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.156+0000 7f08f57fa640 10 client.4445.objecter ms_handle_connect 0x7f08e406df80 2024-03-29T11:40:04.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.164+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08ec050280 con 0x7f08f813f390 2024-03-29T11:40:04.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.164+0000 7f08f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.164+0000 7f08f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:04.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.520+0000 7f08fee31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.520+0000 7f08fee31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.520+0000 7f08fee31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:04.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.520+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08f81401a0 con 0x7f08f813f390 2024-03-29T11:40:04.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.521+0000 7f08f57fa640 1 -- 172.21.15.22:0/2468722243 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f08ec0521b0 con 0x7f08f813f390 2024-03-29T11:40:04.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.521+0000 7f08f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.521+0000 7f08f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:04.523 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:40:04.523 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:04.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.524+0000 7f08fee31640 10 librados: watch_flush enter 2024-03-29T11:40:04.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.524+0000 7f08fee31640 10 librados: watch_flush exit 2024-03-29T11:40:04.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:40:04.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:40:04.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 >> v1:172.21.15.204:6816/3848099522 conn(0x7f08e406df80 legacy=0x7f08e4070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 10 monclient: shutdown 2024-03-29T11:40:04.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:04.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 >> v1:172.21.15.22:6790/0 conn(0x7f08f813f390 legacy=0x7f08f8130f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 shutdown_connections 2024-03-29T11:40:04.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 >> 172.21.15.22:0/2468722243 conn(0x7f08f8066a00 msgr2=0x7f08f80b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:04.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 shutdown_connections 2024-03-29T11:40:04.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 1 -- 172.21.15.22:0/2468722243 wait complete. 2024-03-29T11:40:04.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:04.525+0000 7f08fee31640 1 librados: shutdown 2024-03-29T11:40:04.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:11.351 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.483+0000 7f834644e640 10 monclient: get_monmap_and_config 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.483+0000 7f834644e640 10 monclient: build_initial_monmap 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.483+0000 7f834644e640 10 monclient: monmap: 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:40:11.484377+0000 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:40:11.484377+0000 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:40:11.485 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:11.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.484+0000 7f834644e640 1 Processor -- start 2024-03-29T11:40:11.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.484+0000 7f834644e640 1 -- start start 2024-03-29T11:40:11.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.484+0000 7f834644e640 10 monclient: init 2024-03-29T11:40:11.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 10 monclient(hunting): picked mon.noname-a con 0x7f83401457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 10 monclient(hunting): picked mon.noname-c con 0x7f834013f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 10 monclient(hunting): picked mon.noname-b con 0x7f83400a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400aaef0 con 0x7f83401457f0 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340145bd0 con 0x7f834013f650 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340145eb0 con 0x7f83400a9d70 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f834644e640 10 monclient(hunting): authenticate will time out at 1261.763286s 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833ffff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f83401457f0 0x7f834013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36120/0 (socket says 172.21.15.22:36120) 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833ffff640 1 -- 172.21.15.22:0/1529237494 learned_addr learned my addr 172.21.15.22:0/1529237494 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180125870 0 0) 0x7f83400aaef0 con 0x7f83401457f0 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833effd640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:40:11.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f832c003520 con 0x7f83401457f0 2024-03-29T11:40:11.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2833265649 0 0) 0x7f832c003520 con 0x7f83401457f0 2024-03-29T11:40:11.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.486+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 >> v1:172.21.15.22:6790/0 conn(0x7f83400a9d70 legacy=0x7f834014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 >> v1:172.21.15.204:6789/0 conn(0x7f834013f650 legacy=0x7f83400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 1 monclient: found mon.noname-a 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340146340 con 0x7f83401457f0 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 10 monclient: _finish_auth 0 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f8338001a10 con 0x7f83401457f0 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 10 monclient: dump: 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f834644e640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f834644e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8338001ca0 con 0x7f83401457f0 2024-03-29T11:40:11.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f833effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:11.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f834644e640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:11.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.488+0000 7f834644e640 4 set_mon_vals no callback set 2024-03-29T11:40:11.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f833effd640 1 -- 172.21.15.22:0/1529237494 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f8338002030 con 0x7f83401457f0 2024-03-29T11:40:11.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f833effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f833effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:11.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f833effd640 10 monclient: dump: 2024-03-29T11:40:11.496 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:11.496 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:11.497 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:11.497 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:11.497 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.497 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:11.497 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:11.497 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f834644e640 10 monclient: shutdown 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f834644e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f834644e640 1 -- 172.21.15.22:0/1529237494 >> v1:172.21.15.22:6789/0 conn(0x7f83401457f0 legacy=0x7f834013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f834644e640 1 -- 172.21.15.22:0/1529237494 shutdown_connections 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f834644e640 1 -- 172.21.15.22:0/1529237494 >> 172.21.15.22:0/1529237494 conn(0x7f8340066a00 msgr2=0x7f8340066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f834644e640 1 -- 172.21.15.22:0/1529237494 shutdown_connections 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.489+0000 7f834644e640 1 -- 172.21.15.22:0/1529237494 wait complete. 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.490+0000 7f834644e640 10 monclient: build_initial_monmap 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.490+0000 7f834644e640 10 monclient: monmap: 2024-03-29T11:40:11.498 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:11.499 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:11.499 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:40:11.499 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:40:11.499 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:11.499 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:11.499 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:11.499 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.491+0000 7f834644e640 1 librados: starting msgr at 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.491+0000 7f834644e640 1 librados: starting objecter 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.491+0000 7f834644e640 1 Processor -- start 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.491+0000 7f834644e640 1 -- start start 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.491+0000 7f834644e640 1 librados: setting wanted keys 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.491+0000 7f834644e640 1 librados: calling monclient init 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.491+0000 7f834644e640 10 monclient: init 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.492+0000 7f834644e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.492+0000 7f834644e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:11.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.492+0000 7f834644e640 10 monclient(hunting): picked mon.noname-b con 0x7f83400a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:11.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f834644e640 10 monclient(hunting): picked mon.noname-c con 0x7f834013f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:11.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f834644e640 10 monclient(hunting): picked mon.noname-a con 0x7f83401457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:11.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f834644e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340142770 con 0x7f83401457f0 2024-03-29T11:40:11.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f834644e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340141d60 con 0x7f83400a9d70 2024-03-29T11:40:11.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f834644e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83401447a0 con 0x7f834013f650 2024-03-29T11:40:11.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f834644e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:11.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f834644e640 10 monclient(hunting): authenticate will time out at 1261.770700s 2024-03-29T11:40:11.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f83449eb640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f83401457f0 0x7f83401d7710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36292/0 (socket says 172.21.15.22:36292) 2024-03-29T11:40:11.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f83449eb640 1 -- 172.21.15.22:0/2313608553 learned_addr learned my addr 172.21.15.22:0/2313608553 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:11.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f833cff9640 10 client.?.objecter ms_handle_connect 0x7f83401457f0 2024-03-29T11:40:11.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.493+0000 7f833cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:11.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724624893 0 0) 0x7f8340142770 con 0x7f83401457f0 2024-03-29T11:40:11.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:40:11.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8320004eb0 con 0x7f83401457f0 2024-03-29T11:40:11.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1965844758 0 0) 0x7f8320004eb0 con 0x7f83401457f0 2024-03-29T11:40:11.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8340142770 con 0x7f83401457f0 2024-03-29T11:40:11.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f83280018e0 con 0x7f83401457f0 2024-03-29T11:40:11.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:11.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 10 monclient(hunting): dump: 2024-03-29T11:40:11.512 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:11.512 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:11.513 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:11.513 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:11.513 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.513 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:11.514 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:11.514 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:11.514 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:11.514 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:11.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:11.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3904932684 0 0) 0x7f8340142770 con 0x7f83401457f0 2024-03-29T11:40:11.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.494+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 >> v1:172.21.15.22:6790/0 conn(0x7f834013f650 legacy=0x7f83401d7000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:11.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 >> v1:172.21.15.204:6789/0 conn(0x7f83400a9d70 legacy=0x7f8340142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 1 monclient: found mon.a 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340144930 con 0x7f83401457f0 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 client.?.objecter ms_handle_connect 0x7f83400a9d70 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:11.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 10 monclient: _renew_subs 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83401e2aa0 con 0x7f83401457f0 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 10 monclient: _renew_subs 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83401e2e10 con 0x7f83401457f0 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f834644e640 1 librados: init done 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83280035f0 con 0x7f83401457f0 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f8328003a00 con 0x7f83401457f0 2024-03-29T11:40:11.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833cff9640 10 monclient: dump: 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:11.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.496+0000 7f833e7fc640 4 set_mon_vals no callback set 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.497+0000 7f834644e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.497+0000 7f834644e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.497+0000 7f834644e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.497+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8324005080 con 0x7f83401457f0 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.499+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f8328003b60 con 0x7f83401457f0 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.500+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f8328084a40 con 0x7f83401457f0 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.500+0000 7f833ffff640 10 monclient: get_auth_request con 0x7f832006dc10 auth_method 0 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.500+0000 7f833cff9640 10 client.4434.objecter ms_dispatch 0x7f8340000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.500+0000 7f833cff9640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.500+0000 7f833cff9640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.500+0000 7f833cff9640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.500+0000 7f833cff9640 10 client.4434.objecter ms_handle_connect 0x7f832006dc10 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.505+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8328050320 con 0x7f83401457f0 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.505+0000 7f833cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.505+0000 7f833cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:11.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.833+0000 7f834644e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:11.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.833+0000 7f834644e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:11.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.833+0000 7f834644e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:11.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.833+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8324005330 con 0x7f83401457f0 2024-03-29T11:40:11.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.836+0000 7f833cff9640 1 -- 172.21.15.22:0/2313608553 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f8328052250 con 0x7f83401457f0 2024-03-29T11:40:11.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.836+0000 7f833cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:11.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.836+0000 7f833cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:11.842 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:40:11.842 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:11.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.839+0000 7f834644e640 10 librados: watch_flush enter 2024-03-29T11:40:11.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.839+0000 7f834644e640 10 librados: watch_flush exit 2024-03-29T11:40:11.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.839+0000 7f834644e640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T11:40:11.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.839+0000 7f834644e640 10 client.4434.objecter successfully canceled tick 2024-03-29T11:40:11.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.840+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 >> v1:172.21.15.204:6816/3848099522 conn(0x7f832006dc10 legacy=0x7f83200700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.840+0000 7f834644e640 10 monclient: shutdown 2024-03-29T11:40:11.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.840+0000 7f834644e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:11.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.840+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 >> v1:172.21.15.22:6789/0 conn(0x7f83401457f0 legacy=0x7f83401d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.840+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 shutdown_connections 2024-03-29T11:40:11.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.840+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 >> 172.21.15.22:0/2313608553 conn(0x7f8340066a00 msgr2=0x7f8340066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:11.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.840+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 shutdown_connections 2024-03-29T11:40:11.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.840+0000 7f834644e640 1 -- 172.21.15.22:0/2313608553 wait complete. 2024-03-29T11:40:11.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:11.842+0000 7f834644e640 1 librados: shutdown 2024-03-29T11:40:11.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:18.668 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:18.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.817+0000 7fb380089640 10 monclient: get_monmap_and_config 2024-03-29T11:40:18.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.817+0000 7fb380089640 10 monclient: build_initial_monmap 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.817+0000 7fb380089640 10 monclient: monmap: 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:40:18.819119+0000 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:40:18.819119+0000 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:40:18.821 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:18.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.819+0000 7fb380089640 1 Processor -- start 2024-03-29T11:40:18.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.819+0000 7fb380089640 1 -- start start 2024-03-29T11:40:18.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.819+0000 7fb380089640 10 monclient: init 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 10 monclient(hunting): picked mon.noname-a con 0x7fb3780a9930 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 10 monclient(hunting): picked mon.noname-b con 0x7fb378149dc0 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 10 monclient(hunting): picked mon.noname-c con 0x7fb37814ff40 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3780aae60 con 0x7fb3780a9930 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3780b4310 con 0x7fb37814ff40 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3780afdd0 con 0x7fb378149dc0 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.820+0000 7fb380089640 10 monclient(hunting): authenticate will time out at 1269.097276s 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.823+0000 7fb37d624640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb378149dc0 0x7fb37814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:51821/0 (socket says 172.21.15.22:51821) 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.823+0000 7fb37d624640 1 -- 172.21.15.22:0/887775489 learned_addr learned my addr 172.21.15.22:0/887775489 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:18.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.823+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755540267 0 0) 0x7fb3780afdd0 con 0x7fb378149dc0 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.823+0000 7fb37ce23640 10 monclient(hunting): _negotiate 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.823+0000 7fb37ce23640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.823+0000 7fb37ce23640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.823+0000 7fb37ce23640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.823+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb364003520 con 0x7fb378149dc0 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3201903326 0 0) 0x7fb364003520 con 0x7fb378149dc0 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 >> v1:172.21.15.204:6789/0 conn(0x7fb37814ff40 legacy=0x7fb378152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 >> v1:172.21.15.22:6789/0 conn(0x7fb3780a9930 legacy=0x7fb378143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 10 monclient: _finish_hunting 0 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 1 monclient: found mon.noname-b 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37814a950 con 0x7fb378149dc0 2024-03-29T11:40:18.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 10 monclient: _finish_auth 0 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 10 monclient: _check_auth_tickets 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb368001690 con 0x7fb378149dc0 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 10 monclient: dump: 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:18.831 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb368001860 con 0x7fb378149dc0 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.824+0000 7fb37ce23640 1 -- 172.21.15.22:0/887775489 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb368001cb0 con 0x7fb378149dc0 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb37ce23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb37ce23640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb37ce23640 10 monclient: dump: 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:18.832 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb380089640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb380089640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb380089640 4 set_mon_vals no callback set 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb380089640 10 monclient: shutdown 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb380089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:18.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.825+0000 7fb380089640 1 -- 172.21.15.22:0/887775489 >> v1:172.21.15.22:6790/0 conn(0x7fb378149dc0 legacy=0x7fb37814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.828+0000 7fb380089640 1 -- 172.21.15.22:0/887775489 shutdown_connections 2024-03-29T11:40:18.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.828+0000 7fb380089640 1 -- 172.21.15.22:0/887775489 >> 172.21.15.22:0/887775489 conn(0x7fb3780b35e0 msgr2=0x7fb3780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:18.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.828+0000 7fb380089640 1 -- 172.21.15.22:0/887775489 shutdown_connections 2024-03-29T11:40:18.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.828+0000 7fb380089640 1 -- 172.21.15.22:0/887775489 wait complete. 2024-03-29T11:40:18.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.829+0000 7fb380089640 10 monclient: build_initial_monmap 2024-03-29T11:40:18.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.829+0000 7fb380089640 10 monclient: monmap: 2024-03-29T11:40:18.834 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:18.834 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.829+0000 7fb380089640 1 librados: starting msgr at 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.829+0000 7fb380089640 1 librados: starting objecter 2024-03-29T11:40:18.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.829+0000 7fb380089640 1 Processor -- start 2024-03-29T11:40:18.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.833+0000 7fb380089640 1 -- start start 2024-03-29T11:40:18.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.833+0000 7fb380089640 1 librados: setting wanted keys 2024-03-29T11:40:18.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.833+0000 7fb380089640 1 librados: calling monclient init 2024-03-29T11:40:18.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.833+0000 7fb380089640 10 monclient: init 2024-03-29T11:40:18.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.833+0000 7fb380089640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:18.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.833+0000 7fb380089640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:18.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.836+0000 7fb380089640 10 monclient(hunting): picked mon.noname-b con 0x7fb3780a9930 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:18.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.836+0000 7fb380089640 10 monclient(hunting): picked mon.noname-a con 0x7fb378149dc0 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:18.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.836+0000 7fb37de25640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fb3780a9930 0x7fb378146370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:36365/0 (socket says 172.21.15.22:36365) 2024-03-29T11:40:18.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb380089640 10 monclient(hunting): picked mon.noname-c con 0x7fb37814ff40 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:18.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb37de25640 1 -- 172.21.15.22:0/2294840433 learned_addr learned my addr 172.21.15.22:0/2294840433 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:18.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37814aae0 con 0x7fb378149dc0 2024-03-29T11:40:18.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb378147060 con 0x7fb3780a9930 2024-03-29T11:40:18.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb378147340 con 0x7fb37814ff40 2024-03-29T11:40:18.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb380089640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:18.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb380089640 10 monclient(hunting): authenticate will time out at 1269.114679s 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb36e7fc640 10 client.?.objecter ms_handle_connect 0x7fb3780a9930 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb36e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.837+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778023284 0 0) 0x7fb378147060 con 0x7fb3780a9930 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.838+0000 7fb36e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.838+0000 7fb36e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.838+0000 7fb36e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.838+0000 7fb36e7fc640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.838+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb35c004eb0 con 0x7fb3780a9930 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.838+0000 7fb36e7fc640 10 client.?.objecter ms_handle_connect 0x7fb378149dc0 2024-03-29T11:40:18.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.838+0000 7fb36e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:18.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.839+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174834076 0 0) 0x7fb35c004eb0 con 0x7fb3780a9930 2024-03-29T11:40:18.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.839+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb378147060 con 0x7fb3780a9930 2024-03-29T11:40:18.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.839+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb370001d90 con 0x7fb3780a9930 2024-03-29T11:40:18.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.839+0000 7fb36e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:18.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.839+0000 7fb36e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:18.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.839+0000 7fb36e7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:18.845 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.839+0000 7fb36e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2777643563 0 0) 0x7fb378147060 con 0x7fb3780a9930 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 >> v1:172.21.15.22:6790/0 conn(0x7fb37814ff40 legacy=0x7fb378147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 >> v1:172.21.15.22:6789/0 conn(0x7fb378149dc0 legacy=0x7fb378131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:18.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 1 monclient: found mon.b 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37806e8b0 con 0x7fb3780a9930 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 10 monclient: _renew_subs 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb378145e50 con 0x7fb3780a9930 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 10 monclient: _renew_subs 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:40:18.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb378072660 con 0x7fb3780a9930 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb380089640 1 librados: init done 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb370001ef0 con 0x7fb3780a9930 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb370003de0 con 0x7fb3780a9930 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.840+0000 7fb36e7fc640 10 monclient: dump: 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:18.848 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.841+0000 7fb36f7fe640 4 set_mon_vals no callback set 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.841+0000 7fb380089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.842+0000 7fb380089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.842+0000 7fb380089640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.842+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fb370003f40 con 0x7fb3780a9930 2024-03-29T11:40:18.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.842+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb378075f10 con 0x7fb3780a9930 2024-03-29T11:40:18.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.843+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fb370084f40 con 0x7fb3780a9930 2024-03-29T11:40:18.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.843+0000 7fb36e7fc640 10 client.4414.objecter ms_dispatch 0x7fb378000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:18.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.843+0000 7fb36e7fc640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:18.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.843+0000 7fb36e7fc640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:18.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.843+0000 7fb36e7fc640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T11:40:18.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.844+0000 7fb37d624640 10 monclient: get_auth_request con 0x7fb35c06dc70 auth_method 0 2024-03-29T11:40:18.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.844+0000 7fb36e7fc640 10 client.4414.objecter ms_handle_connect 0x7fb35c06dc70 2024-03-29T11:40:18.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.851+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb370050820 con 0x7fb3780a9930 2024-03-29T11:40:18.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.851+0000 7fb36e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:18.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:18.851+0000 7fb36e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:19.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.181+0000 7fb380089640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.181+0000 7fb380089640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.181+0000 7fb380089640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:40:19.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.181+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3781477d0 con 0x7fb3780a9930 2024-03-29T11:40:19.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.182+0000 7fb36e7fc640 1 -- 172.21.15.22:0/2294840433 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fb370052750 con 0x7fb3780a9930 2024-03-29T11:40:19.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.182+0000 7fb36e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.182+0000 7fb36e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:19.186 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:40:19.186 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:19.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.185+0000 7fb380089640 10 librados: watch_flush enter 2024-03-29T11:40:19.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 10 librados: watch_flush exit 2024-03-29T11:40:19.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T11:40:19.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 10 client.4414.objecter successfully canceled tick 2024-03-29T11:40:19.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 >> v1:172.21.15.204:6816/3848099522 conn(0x7fb35c06dc70 legacy=0x7fb35c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 10 monclient: shutdown 2024-03-29T11:40:19.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:19.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 >> v1:172.21.15.204:6789/0 conn(0x7fb3780a9930 legacy=0x7fb378146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 shutdown_connections 2024-03-29T11:40:19.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 >> 172.21.15.22:0/2294840433 conn(0x7fb3780b35e0 msgr2=0x7fb378140b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:19.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 shutdown_connections 2024-03-29T11:40:19.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.186+0000 7fb380089640 1 -- 172.21.15.22:0/2294840433 wait complete. 2024-03-29T11:40:19.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:19.187+0000 7fb380089640 1 librados: shutdown 2024-03-29T11:40:19.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:26.014 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:26.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.162+0000 7fd3db12e640 10 monclient: get_monmap_and_config 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.162+0000 7fd3db12e640 10 monclient: build_initial_monmap 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.162+0000 7fd3db12e640 10 monclient: monmap: 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:40:26.163326+0000 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:40:26.163326+0000 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:40:26.165 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:40:26.166 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:26.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.163+0000 7fd3db12e640 1 Processor -- start 2024-03-29T11:40:26.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.163+0000 7fd3db12e640 1 -- start start 2024-03-29T11:40:26.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.163+0000 7fd3db12e640 10 monclient: init 2024-03-29T11:40:26.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:26.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:26.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3cc034850 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:26.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3cc02a640 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:26.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3cc0327d0 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:26.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc02c820 con 0x7fd3cc02a640 2024-03-29T11:40:26.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc02a120 con 0x7fd3cc0327d0 2024-03-29T11:40:26.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc0c75b0 con 0x7fd3cc034850 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.164+0000 7fd3db12e640 10 monclient(hunting): authenticate will time out at 1276.441678s 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.165+0000 7fd3da12c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fd3cc034850 0x7fd3cc0c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47350/0 (socket says 172.21.15.22:47350) 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.165+0000 7fd3da12c640 1 -- 172.21.15.22:0/3388768038 learned_addr learned my addr 172.21.15.22:0/3388768038 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474834874 0 0) 0x7fd3cc0c75b0 con 0x7fd3cc034850 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c0003500 con 0x7fd3cc034850 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054643714 0 0) 0x7fd3c0003500 con 0x7fd3cc034850 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 >> v1:172.21.15.204:6789/0 conn(0x7fd3cc0327d0 legacy=0x7fd3cc0cf240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:26.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.166+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 >> v1:172.21.15.22:6789/0 conn(0x7fd3cc02a640 legacy=0x7fd3cc0320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:26.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 10 monclient: _finish_hunting 0 2024-03-29T11:40:26.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 1 monclient: found mon.noname-b 2024-03-29T11:40:26.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:26.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:40:26.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc033110 con 0x7fd3cc034850 2024-03-29T11:40:26.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 10 monclient: _finish_auth 0 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 10 monclient: _check_auth_tickets 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd3c4004cb0 con 0x7fd3cc034850 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3d912a640 10 monclient: dump: 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.171 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:26.172 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:26.172 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:26.172 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:26.172 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:26.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3db12e640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:40:26.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.168+0000 7fd3db12e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.171+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3c4001250 con 0x7fd3cc034850 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.171+0000 7fd3d912a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.171+0000 7fd3d912a640 1 -- 172.21.15.22:0/3388768038 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd3c40015e0 con 0x7fd3cc034850 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.171+0000 7fd3d912a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.171+0000 7fd3d912a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.171+0000 7fd3d912a640 10 monclient: dump: 2024-03-29T11:40:26.173 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:26.174 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.171+0000 7fd3db12e640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.171+0000 7fd3db12e640 4 set_mon_vals no callback set 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.172+0000 7fd3db12e640 10 monclient: shutdown 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.172+0000 7fd3db12e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:26.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.172+0000 7fd3db12e640 1 -- 172.21.15.22:0/3388768038 >> v1:172.21.15.22:6790/0 conn(0x7fd3cc034850 legacy=0x7fd3cc0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.176+0000 7fd3db12e640 1 -- 172.21.15.22:0/3388768038 shutdown_connections 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.176+0000 7fd3db12e640 1 -- 172.21.15.22:0/3388768038 >> 172.21.15.22:0/3388768038 conn(0x7fd3cc0058f0 msgr2=0x7fd3cc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.176+0000 7fd3db12e640 1 -- 172.21.15.22:0/3388768038 shutdown_connections 2024-03-29T11:40:26.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.176+0000 7fd3db12e640 1 -- 172.21.15.22:0/3388768038 wait complete. 2024-03-29T11:40:26.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.177+0000 7fd3db12e640 10 monclient: build_initial_monmap 2024-03-29T11:40:26.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.177+0000 7fd3db12e640 10 monclient: monmap: 2024-03-29T11:40:26.179 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:26.179 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:26.179 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:40:26.179 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:40:26.179 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:26.179 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:26.180 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:26.180 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:40:26.180 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:40:26.180 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:26.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.177+0000 7fd3db12e640 1 librados: starting msgr at 2024-03-29T11:40:26.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.177+0000 7fd3db12e640 1 librados: starting objecter 2024-03-29T11:40:26.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 1 Processor -- start 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 1 -- start start 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 1 librados: setting wanted keys 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 1 librados: calling monclient init 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 10 monclient: init 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3cc02a640 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3cc0327d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3cc034850 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc0bc9a0 con 0x7fd3cc02a640 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc0bb660 con 0x7fd3cc034850 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc14f9b0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:26.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.178+0000 7fd3db12e640 10 monclient(hunting): authenticate will time out at 1276.455935s 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3d992b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fd3cc0327d0 0x7fd3cc168840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49308/0 (socket says 172.21.15.22:49308) 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3d992b640 1 -- 172.21.15.22:0/2426563624 learned_addr learned my addr 172.21.15.22:0/2426563624 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3caffd640 10 client.?.objecter ms_handle_connect 0x7fd3cc0327d0 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361112684 0 0) 0x7fd3cc14f9b0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3caffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.179+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3b8004eb0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214820253 0 0) 0x7fd3b8004eb0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3cc14f9b0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd3d404fdd0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 10 monclient(hunting): dump: 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 757182298 0 0) 0x7fd3cc14f9b0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.180+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 >> v1:172.21.15.204:6789/0 conn(0x7fd3cc034850 legacy=0x7fd3cc16cbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:26.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.183+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 >> v1:172.21.15.22:6789/0 conn(0x7fd3cc02a640 legacy=0x7fd3cc0ba560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:26.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:26.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3caffd640 1 monclient: found mon.c 2024-03-29T11:40:26.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:26.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:26.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc0bb2c0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3caffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 10 monclient: _renew_subs 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 1 -- 172.21.15.22:0/2426563624 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3cc0bcd50 con 0x7fd3cc0327d0 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 10 monclient: _renew_subs 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 1 -- 172.21.15.22:0/2426563624 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3cc169130 con 0x7fd3cc0327d0 2024-03-29T11:40:26.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.184+0000 7fd3db12e640 1 librados: init done 2024-03-29T11:40:26.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.185+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3d404f400 con 0x7fd3cc0327d0 2024-03-29T11:40:26.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.185+0000 7fd3caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:26.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.185+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd3d4075f10 con 0x7fd3cc0327d0 2024-03-29T11:40:26.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.185+0000 7fd3caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.185+0000 7fd3caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:26.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.185+0000 7fd3caffd640 10 monclient: dump: 2024-03-29T11:40:26.191 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:26.191 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:26.191 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:26.191 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:26.191 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.185+0000 7fd3d8929640 4 set_mon_vals no callback set 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.187+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fd3d4067bf0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.189+0000 7fd3c8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.189+0000 7fd3c8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.189+0000 7fd3c8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.189+0000 7fd3c8ff9640 1 -- 172.21.15.22:0/2426563624 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3cc0c4b10 con 0x7fd3cc0327d0 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.190+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fd3d40ff600 con 0x7fd3cc0327d0 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.190+0000 7fd3caffd640 10 client.4454.objecter ms_dispatch 0x7fd3d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.190+0000 7fd3caffd640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.190+0000 7fd3caffd640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.190+0000 7fd3caffd640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.191+0000 7fd3da12c640 10 monclient: get_auth_request con 0x7fd3b806db40 auth_method 0 2024-03-29T11:40:26.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.193+0000 7fd3caffd640 10 client.4454.objecter ms_handle_connect 0x7fd3b806db40 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.199+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3d40caf40 con 0x7fd3cc0327d0 2024-03-29T11:40:26.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.199+0000 7fd3caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.199+0000 7fd3caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:26.536 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:40:26.536 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:26.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.532+0000 7fd3c8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.532+0000 7fd3c8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.532+0000 7fd3c8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.532+0000 7fd3c8ff9640 1 -- 172.21.15.22:0/2426563624 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3cc0bb7f0 con 0x7fd3cc0327d0 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.534+0000 7fd3caffd640 1 -- 172.21.15.22:0/2426563624 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fd3d40cce70 con 0x7fd3cc0327d0 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.534+0000 7fd3caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.534+0000 7fd3caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:26.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.536+0000 7fd3c8ff9640 10 librados: watch_flush enter 2024-03-29T11:40:26.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.536+0000 7fd3c8ff9640 10 librados: watch_flush exit 2024-03-29T11:40:26.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.537+0000 7fd3c8ff9640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T11:40:26.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.537+0000 7fd3c8ff9640 10 client.4454.objecter successfully canceled tick 2024-03-29T11:40:26.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.538+0000 7fd3c8ff9640 1 -- 172.21.15.22:0/2426563624 >> v1:172.21.15.204:6816/3848099522 conn(0x7fd3b806db40 legacy=0x7fd3b8070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.538+0000 7fd3c8ff9640 10 monclient: shutdown 2024-03-29T11:40:26.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.538+0000 7fd3c8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:26.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.538+0000 7fd3c8ff9640 1 -- 172.21.15.22:0/2426563624 >> v1:172.21.15.22:6790/0 conn(0x7fd3cc0327d0 legacy=0x7fd3cc168840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.539+0000 7fd3c8ff9640 1 -- 172.21.15.22:0/2426563624 shutdown_connections 2024-03-29T11:40:26.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.539+0000 7fd3c8ff9640 1 -- 172.21.15.22:0/2426563624 >> 172.21.15.22:0/2426563624 conn(0x7fd3cc0058f0 msgr2=0x7fd3cc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:26.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.539+0000 7fd3c8ff9640 1 -- 172.21.15.22:0/2426563624 shutdown_connections 2024-03-29T11:40:26.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.540+0000 7fd3c8ff9640 1 -- 172.21.15.22:0/2426563624 wait complete. 2024-03-29T11:40:26.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:26.540+0000 7fd3c8ff9640 1 librados: shutdown 2024-03-29T11:40:26.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:33.365 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:33.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.518+0000 7f47e215d640 10 monclient: get_monmap_and_config 2024-03-29T11:40:33.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.518+0000 7f47e215d640 10 monclient: build_initial_monmap 2024-03-29T11:40:33.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.518+0000 7f47e215d640 10 monclient: monmap: 2024-03-29T11:40:33.520 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:40:33.519630+0000 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:40:33.519630+0000 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:40:33.521 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:33.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.520+0000 7f47e215d640 1 Processor -- start 2024-03-29T11:40:33.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.520+0000 7f47e215d640 1 -- start start 2024-03-29T11:40:33.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.520+0000 7f47e215d640 10 monclient: init 2024-03-29T11:40:33.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.520+0000 7f47e215d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.520+0000 7f47e215d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.520+0000 7f47e215d640 10 monclient(hunting): picked mon.noname-c con 0x7f47dc0b1f60 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.520+0000 7f47e215d640 10 monclient(hunting): picked mon.noname-a con 0x7f47dc071eb0 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.521+0000 7f47e215d640 10 monclient(hunting): picked mon.noname-b con 0x7f47dc143d10 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.521+0000 7f47e215d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc06ef80 con 0x7f47dc071eb0 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.521+0000 7f47e215d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc06f260 con 0x7f47dc0b1f60 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.521+0000 7f47e215d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc0b40c0 con 0x7f47dc143d10 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.521+0000 7f47e215d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:33.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.521+0000 7f47e215d640 10 monclient(hunting): authenticate will time out at 1283.798048s 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.524+0000 7f47dbfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f47dc143d10 0x7f47dc146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56817/0 (socket says 172.21.15.22:56817) 2024-03-29T11:40:33.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.524+0000 7f47dbfff640 1 -- 172.21.15.22:0/490334494 learned_addr learned my addr 172.21.15.22:0/490334494 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:33.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.527+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155858605 0 0) 0x7f47dc0b40c0 con 0x7f47dc143d10 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.527+0000 7f47da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.527+0000 7f47da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.527+0000 7f47da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.527+0000 7f47da7fc640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:40:33.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.528+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c4003520 con 0x7f47dc143d10 2024-03-29T11:40:33.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.528+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955833771 0 0) 0x7f47dc06ef80 con 0x7f47dc071eb0 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.528+0000 7f47da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.528+0000 7f47da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.528+0000 7f47da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.528+0000 7f47da7fc640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.528+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47dc0b40c0 con 0x7f47dc071eb0 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294346815 0 0) 0x7f47dc0b40c0 con 0x7f47dc071eb0 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 >> v1:172.21.15.22:6790/0 conn(0x7f47dc143d10 legacy=0x7f47dc146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 >> v1:172.21.15.204:6789/0 conn(0x7f47dc0b1f60 legacy=0x7f47dc143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 1 monclient: found mon.noname-a 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:40:33.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc155d70 con 0x7f47dc071eb0 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f47d00015a0 con 0x7f47dc071eb0 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: dump: 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:33.535 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47e215d640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47e215d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47d0001700 con 0x7f47dc071eb0 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47e215d640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47e215d640 4 set_mon_vals no callback set 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 1 -- 172.21.15.22:0/490334494 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f47d0001b50 con 0x7f47dc071eb0 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.529+0000 7f47da7fc640 10 monclient: dump: 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:33.536 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.530+0000 7f47e215d640 10 monclient: shutdown 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.530+0000 7f47e215d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:33.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.530+0000 7f47e215d640 1 -- 172.21.15.22:0/490334494 >> v1:172.21.15.22:6789/0 conn(0x7f47dc071eb0 legacy=0x7f47dc146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.531+0000 7f47e215d640 1 -- 172.21.15.22:0/490334494 shutdown_connections 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.531+0000 7f47e215d640 1 -- 172.21.15.22:0/490334494 >> 172.21.15.22:0/490334494 conn(0x7f47dc066a00 msgr2=0x7f47dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.532+0000 7f47e215d640 1 -- 172.21.15.22:0/490334494 shutdown_connections 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.533+0000 7f47e215d640 1 -- 172.21.15.22:0/490334494 wait complete. 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.533+0000 7f47e215d640 10 monclient: build_initial_monmap 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.533+0000 7f47e215d640 10 monclient: monmap: 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:33.538 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.534+0000 7f47e215d640 1 librados: starting msgr at 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.534+0000 7f47e215d640 1 librados: starting objecter 2024-03-29T11:40:33.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.534+0000 7f47e215d640 1 Processor -- start 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.535+0000 7f47e215d640 1 -- start start 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.535+0000 7f47e215d640 1 librados: setting wanted keys 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.535+0000 7f47e215d640 1 librados: calling monclient init 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.535+0000 7f47e215d640 10 monclient: init 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 10 monclient(hunting): picked mon.noname-b con 0x7f47dc0b1f60 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 10 monclient(hunting): picked mon.noname-c con 0x7f47dc143d10 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 10 monclient(hunting): picked mon.noname-a con 0x7f47dc154260 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc0ac100 con 0x7f47dc154260 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc0ac3e0 con 0x7f47dc0b1f60 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc0ab4c0 con 0x7f47dc143d10 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:33.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.536+0000 7f47e215d640 10 monclient(hunting): authenticate will time out at 1283.813519s 2024-03-29T11:40:33.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.540+0000 7f47dbfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f47dc154260 0x7f47dc1ef390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46223/0 (socket says 172.21.15.22:46223) 2024-03-29T11:40:33.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.540+0000 7f47dbfff640 1 -- 172.21.15.22:0/1597156780 learned_addr learned my addr 172.21.15.22:0/1597156780 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 client.?.objecter ms_handle_connect 0x7f47dc154260 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 client.?.objecter ms_handle_connect 0x7f47dc0b1f60 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688120582 0 0) 0x7f47dc0ac100 con 0x7f47dc154260 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c8004eb0 con 0x7f47dc154260 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059159529 0 0) 0x7f47dc0ac3e0 con 0x7f47dc0b1f60 2024-03-29T11:40:33.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.542+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47dc0ac100 con 0x7f47dc0b1f60 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 696045699 0 0) 0x7f47c8004eb0 con 0x7f47dc154260 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47dc0ac3e0 con 0x7f47dc154260 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f47d4001b90 con 0x7f47dc154260 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 10 monclient(hunting): dump: 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:33.546 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:33.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1096582387 0 0) 0x7f47dc0ac3e0 con 0x7f47dc154260 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 >> v1:172.21.15.22:6790/0 conn(0x7f47dc143d10 legacy=0x7f47dc13fc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.543+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 >> v1:172.21.15.204:6789/0 conn(0x7f47dc0b1f60 legacy=0x7f47dc153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47c3fff640 1 monclient: found mon.a 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47c3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc0aa2d0 con 0x7f47dc154260 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 10 monclient: _renew_subs 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc0aa580 con 0x7f47dc154260 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 10 monclient: _renew_subs 2024-03-29T11:40:33.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:33.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc140540 con 0x7f47dc154260 2024-03-29T11:40:33.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.544+0000 7f47e215d640 1 librados: init done 2024-03-29T11:40:33.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.547+0000 7f47e215d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.547+0000 7f47e215d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.547+0000 7f47e215d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:33.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.547+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc0ab310 con 0x7f47dc154260 2024-03-29T11:40:33.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.548+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47d40020a0 con 0x7f47dc154260 2024-03-29T11:40:33.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.548+0000 7f47c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:33.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.548+0000 7f47d97fa640 4 set_mon_vals no callback set 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.551+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f47d4004d00 con 0x7f47dc154260 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.551+0000 7f47c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.551+0000 7f47c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.551+0000 7f47c3fff640 10 monclient: dump: 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.553 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.551+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f47d401ad50 con 0x7f47dc154260 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.552+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f47d4084d70 con 0x7f47dc154260 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.552+0000 7f47c3fff640 10 client.4446.objecter ms_dispatch 0x7f47dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.552+0000 7f47c3fff640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:33.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.552+0000 7f47c3fff640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:33.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.552+0000 7f47c3fff640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-29T11:40:33.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.556+0000 7f47db7fe640 10 monclient: get_auth_request con 0x7f47c806dbb0 auth_method 0 2024-03-29T11:40:33.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.556+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47d4050500 con 0x7f47dc154260 2024-03-29T11:40:33.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.556+0000 7f47c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.556+0000 7f47c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:33.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.556+0000 7f47c3fff640 10 client.4446.objecter ms_handle_connect 0x7f47c806dbb0 2024-03-29T11:40:33.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.883+0000 7f47e215d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:33.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.883+0000 7f47e215d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:33.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.883+0000 7f47e215d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:33.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.883+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47dc1407e0 con 0x7f47dc154260 2024-03-29T11:40:33.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.888+0000 7f47c3fff640 1 -- 172.21.15.22:0/1597156780 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f47dc1407e0 con 0x7f47dc154260 2024-03-29T11:40:33.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.888+0000 7f47c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:33.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.888+0000 7f47c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:33.891 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:40:33.891 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:33.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.893+0000 7f47e215d640 10 librados: watch_flush enter 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.893+0000 7f47e215d640 10 librados: watch_flush exit 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.893+0000 7f47e215d640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.893+0000 7f47e215d640 10 client.4446.objecter successfully canceled tick 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.893+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 >> v1:172.21.15.204:6816/3848099522 conn(0x7f47c806dbb0 legacy=0x7f47c8070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.893+0000 7f47e215d640 10 monclient: shutdown 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.893+0000 7f47e215d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.893+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 >> v1:172.21.15.22:6789/0 conn(0x7f47dc154260 legacy=0x7f47dc1ef390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.894+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 shutdown_connections 2024-03-29T11:40:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.894+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 >> 172.21.15.22:0/1597156780 conn(0x7f47dc066a00 msgr2=0x7f47dc146960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:33.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.894+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 shutdown_connections 2024-03-29T11:40:33.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.894+0000 7f47e215d640 1 -- 172.21.15.22:0/1597156780 wait complete. 2024-03-29T11:40:33.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:33.894+0000 7f47e215d640 1 librados: shutdown 2024-03-29T11:40:33.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:40.719 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.871+0000 7f5b79c8f640 10 monclient: get_monmap_and_config 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.871+0000 7f5b79c8f640 10 monclient: build_initial_monmap 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.871+0000 7f5b79c8f640 10 monclient: monmap: 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:40:40.872467+0000 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:40:40.872467+0000 2024-03-29T11:40:40.875 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.872+0000 7f5b79c8f640 1 Processor -- start 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.872+0000 7f5b79c8f640 1 -- start start 2024-03-29T11:40:40.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.872+0000 7f5b79c8f640 10 monclient: init 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 10 monclient(hunting): picked mon.noname-a con 0x7f5b740b4100 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 10 monclient(hunting): picked mon.noname-b con 0x7f5b7413ee90 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 10 monclient(hunting): picked mon.noname-c con 0x7f5b740af900 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b740afd10 con 0x7f5b740b4100 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b740af6f0 con 0x7f5b740af900 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b740aaf60 con 0x7f5b7413ee90 2024-03-29T11:40:40.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:40.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.873+0000 7f5b79c8f640 10 monclient(hunting): authenticate will time out at 1291.150558s 2024-03-29T11:40:40.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.876+0000 7f5b72ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5b7413ee90 0x7f5b740adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45868/0 (socket says 172.21.15.22:45868) 2024-03-29T11:40:40.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.876+0000 7f5b72ffd640 1 -- 172.21.15.22:0/2996852795 learned_addr learned my addr 172.21.15.22:0/2996852795 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:40.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.877+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237297008 0 0) 0x7f5b740aaf60 con 0x7f5b7413ee90 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.877+0000 7f5b727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.877+0000 7f5b727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.877+0000 7f5b727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.877+0000 7f5b727fc640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:40:40.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.877+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b5c003520 con 0x7f5b7413ee90 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72113378 0 0) 0x7f5b740afd10 con 0x7f5b740b4100 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b740aaf60 con 0x7f5b740b4100 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888531194 0 0) 0x7f5b740af6f0 con 0x7f5b740af900 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b5c0039d0 con 0x7f5b740af900 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1217732302 0 0) 0x7f5b5c0039d0 con 0x7f5b740af900 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.878+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 >> v1:172.21.15.22:6790/0 conn(0x7f5b7413ee90 legacy=0x7f5b740adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 >> v1:172.21.15.22:6789/0 conn(0x7f5b740b4100 legacy=0x7f5b740a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 1 monclient: found mon.noname-c 2024-03-29T11:40:40.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7413e970 con 0x7f5b740af900 2024-03-29T11:40:40.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b6c0018f0 con 0x7f5b740af900 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:40.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: dump: 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b79c8f640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b79c8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b6c001ad0 con 0x7f5b740af900 2024-03-29T11:40:40.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 1 -- 172.21.15.22:0/2996852795 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b6c001f20 con 0x7f5b740af900 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b727fc640 10 monclient: dump: 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:40.887 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b79c8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.879+0000 7f5b79c8f640 4 set_mon_vals no callback set 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.880+0000 7f5b79c8f640 10 monclient: shutdown 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.880+0000 7f5b79c8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.880+0000 7f5b79c8f640 1 -- 172.21.15.22:0/2996852795 >> v1:172.21.15.204:6789/0 conn(0x7f5b740af900 legacy=0x7f5b7413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.880+0000 7f5b79c8f640 1 -- 172.21.15.22:0/2996852795 shutdown_connections 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.880+0000 7f5b79c8f640 1 -- 172.21.15.22:0/2996852795 >> 172.21.15.22:0/2996852795 conn(0x7f5b740b35e0 msgr2=0x7f5b740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.881+0000 7f5b79c8f640 1 -- 172.21.15.22:0/2996852795 shutdown_connections 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.881+0000 7f5b79c8f640 1 -- 172.21.15.22:0/2996852795 wait complete. 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.881+0000 7f5b79c8f640 10 monclient: build_initial_monmap 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.881+0000 7f5b79c8f640 10 monclient: monmap: 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:40.888 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.882+0000 7f5b79c8f640 1 librados: starting msgr at 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.882+0000 7f5b79c8f640 1 librados: starting objecter 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.882+0000 7f5b79c8f640 1 Processor -- start 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.882+0000 7f5b79c8f640 1 -- start start 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.882+0000 7f5b79c8f640 1 librados: setting wanted keys 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.882+0000 7f5b79c8f640 1 librados: calling monclient init 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.882+0000 7f5b79c8f640 10 monclient: init 2024-03-29T11:40:40.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:40.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 10 monclient(hunting): picked mon.noname-c con 0x7f5b740af900 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:40.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 10 monclient(hunting): picked mon.noname-a con 0x7f5b740b4100 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:40.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 10 monclient(hunting): picked mon.noname-b con 0x7f5b7413ee90 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:40.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b74140440 con 0x7f5b740b4100 2024-03-29T11:40:40.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b74141ce0 con 0x7f5b7413ee90 2024-03-29T11:40:40.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b74141fc0 con 0x7f5b740af900 2024-03-29T11:40:40.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.883+0000 7f5b79c8f640 10 monclient(hunting): authenticate will time out at 1291.160733s 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b737fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5b740af900 0x7f5b7406e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50730/0 (socket says 172.21.15.22:50730) 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b737fe640 1 -- 172.21.15.22:0/388710960 learned_addr learned my addr 172.21.15.22:0/388710960 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 client.?.objecter ms_handle_connect 0x7f5b740af900 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 client.?.objecter ms_handle_connect 0x7f5b7413ee90 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746085503 0 0) 0x7f5b74141fc0 con 0x7f5b740af900 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b60004eb0 con 0x7f5b740af900 2024-03-29T11:40:40.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377287374 0 0) 0x7f5b74141ce0 con 0x7f5b7413ee90 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.886+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b74141fc0 con 0x7f5b7413ee90 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 10 client.?.objecter ms_handle_connect 0x7f5b740b4100 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2421832285 0 0) 0x7f5b60004eb0 con 0x7f5b740af900 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b74141ce0 con 0x7f5b740af900 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b680019f0 con 0x7f5b740af900 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 10 monclient(hunting): dump: 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:40.897 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.887+0000 7f5b5bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 305720504 0 0) 0x7f5b74141ce0 con 0x7f5b740af900 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 >> v1:172.21.15.204:6789/0 conn(0x7f5b7413ee90 legacy=0x7f5b741e7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 >> v1:172.21.15.22:6789/0 conn(0x7f5b740b4100 legacy=0x7f5b7412e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:40.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 1 monclient: found mon.c 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b741422a0 con 0x7f5b740af900 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 10 monclient: _finish_auth 0 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 10 monclient: _renew_subs 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b74140690 con 0x7f5b740af900 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 10 monclient: _renew_subs 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:40.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7413feb0 con 0x7f5b740af900 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.888+0000 7f5b79c8f640 1 librados: init done 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.890+0000 7f5b79c8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.890+0000 7f5b79c8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.890+0000 7f5b79c8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.890+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b74075f10 con 0x7f5b740af900 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.891+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b68001c50 con 0x7f5b740af900 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.891+0000 7f5b5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.891+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b68003a40 con 0x7f5b740af900 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.891+0000 7f5b5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.891+0000 7f5b5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:40.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.891+0000 7f5b5bfff640 10 monclient: dump: 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:40.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.891+0000 7f5b71ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.892+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f5b68003c60 con 0x7f5b740af900 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.893+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f5b68084a40 con 0x7f5b740af900 2024-03-29T11:40:40.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.893+0000 7f5b5bfff640 10 client.4463.objecter ms_dispatch 0x7f5b74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:40.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.893+0000 7f5b5bfff640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:40.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.893+0000 7f5b5bfff640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:40.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.893+0000 7f5b5bfff640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:40:40.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.894+0000 7f5b72ffd640 10 monclient: get_auth_request con 0x7f5b6006dbd0 auth_method 0 2024-03-29T11:40:40.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.896+0000 7f5b5bfff640 10 client.4463.objecter ms_handle_connect 0x7f5b6006dbd0 2024-03-29T11:40:40.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.901+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b68050320 con 0x7f5b740af900 2024-03-29T11:40:40.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.901+0000 7f5b5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:40.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:40.901+0000 7f5b5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:41.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.241+0000 7f5b79c8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.241+0000 7f5b79c8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.241+0000 7f5b79c8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:41.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.241+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b740ac8b0 con 0x7f5b740af900 2024-03-29T11:40:41.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.243+0000 7f5b5bfff640 1 -- 172.21.15.22:0/388710960 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f5b68052250 con 0x7f5b740af900 2024-03-29T11:40:41.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.243+0000 7f5b5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.243+0000 7f5b5bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:41.247 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:40:41.247 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:41.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.246+0000 7f5b79c8f640 10 librados: watch_flush enter 2024-03-29T11:40:41.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.247+0000 7f5b79c8f640 10 librados: watch_flush exit 2024-03-29T11:40:41.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.247+0000 7f5b79c8f640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:40:41.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.247+0000 7f5b79c8f640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:40:41.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.247+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 >> v1:172.21.15.204:6816/3848099522 conn(0x7f5b6006dbd0 legacy=0x7f5b60070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.247+0000 7f5b79c8f640 10 monclient: shutdown 2024-03-29T11:40:41.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.247+0000 7f5b79c8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:41.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.247+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 >> v1:172.21.15.22:6790/0 conn(0x7f5b740af900 legacy=0x7f5b7406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.249+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 shutdown_connections 2024-03-29T11:40:41.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.249+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 >> 172.21.15.22:0/388710960 conn(0x7f5b740b35e0 msgr2=0x7f5b740abc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:41.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.250+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 shutdown_connections 2024-03-29T11:40:41.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.250+0000 7f5b79c8f640 1 -- 172.21.15.22:0/388710960 wait complete. 2024-03-29T11:40:41.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:41.250+0000 7f5b79c8f640 1 librados: shutdown 2024-03-29T11:40:41.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:48.077 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.229+0000 7f08c23cf640 10 monclient: get_monmap_and_config 2024-03-29T11:40:48.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.229+0000 7f08c23cf640 10 monclient: build_initial_monmap 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.229+0000 7f08c23cf640 10 monclient: monmap: 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:40:48.231137+0000 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:40:48.231137+0000 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:40:48.233 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.231+0000 7f08c23cf640 1 Processor -- start 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.231+0000 7f08c23cf640 1 -- start start 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.231+0000 7f08c23cf640 10 monclient: init 2024-03-29T11:40:48.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 10 monclient(hunting): picked mon.noname-b con 0x7f08bc0b0860 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 10 monclient(hunting): picked mon.noname-a con 0x7f08bc13ef30 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 10 monclient(hunting): picked mon.noname-c con 0x7f08bc0af8e0 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc0b0c40 con 0x7f08bc13ef30 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc0aed10 con 0x7f08bc0af8e0 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc0afd50 con 0x7f08bc0b0860 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.232+0000 7f08c23cf640 10 monclient(hunting): authenticate will time out at 1298.509534s 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.235+0000 7f08bb7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f08bc13ef30 0x7f08bc0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:33364/0 (socket says 172.21.15.22:33364) 2024-03-29T11:40:48.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.235+0000 7f08bb7fe640 1 -- 172.21.15.22:0/1910182077 learned_addr learned my addr 172.21.15.22:0/1910182077 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.235+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481584256 0 0) 0x7f08bc0afd50 con 0x7f08bc0b0860 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.235+0000 7f08baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.235+0000 7f08baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.235+0000 7f08baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.235+0000 7f08baffd640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.236+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a4003520 con 0x7f08bc0b0860 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.236+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152309756 0 0) 0x7f08bc0b0c40 con 0x7f08bc13ef30 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.236+0000 7f08baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.236+0000 7f08baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.236+0000 7f08baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.236+0000 7f08baffd640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:40:48.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.236+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08bc0afd50 con 0x7f08bc13ef30 2024-03-29T11:40:48.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156162183 0 0) 0x7f08bc0afd50 con 0x7f08bc13ef30 2024-03-29T11:40:48.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 >> v1:172.21.15.22:6790/0 conn(0x7f08bc0b0860 legacy=0x7f08bc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 >> v1:172.21.15.204:6789/0 conn(0x7f08bc0af8e0 legacy=0x7f08bc149a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 1 monclient: found mon.noname-a 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0ab1b0 con 0x7f08bc13ef30 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08ac004c90 con 0x7f08bc13ef30 2024-03-29T11:40:48.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.237+0000 7f08baffd640 10 monclient: dump: 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:48.246 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08c23cf640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08c23cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08baffd640 1 -- 172.21.15.22:0/1910182077 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08ac004df0 con 0x7f08bc13ef30 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08c23cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08c23cf640 4 set_mon_vals no callback set 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08c23cf640 10 monclient: shutdown 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08c23cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.238+0000 7f08c23cf640 1 -- 172.21.15.22:0/1910182077 >> v1:172.21.15.22:6789/0 conn(0x7f08bc13ef30 legacy=0x7f08bc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.239+0000 7f08c23cf640 1 -- 172.21.15.22:0/1910182077 shutdown_connections 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.239+0000 7f08c23cf640 1 -- 172.21.15.22:0/1910182077 >> 172.21.15.22:0/1910182077 conn(0x7f08bc06b660 msgr2=0x7f08bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.240+0000 7f08c23cf640 1 -- 172.21.15.22:0/1910182077 shutdown_connections 2024-03-29T11:40:48.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.240+0000 7f08c23cf640 1 -- 172.21.15.22:0/1910182077 wait complete. 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.241+0000 7f08c23cf640 10 monclient: build_initial_monmap 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.241+0000 7f08c23cf640 10 monclient: monmap: 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:40:48.253 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.241+0000 7f08c23cf640 1 librados: starting msgr at 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.241+0000 7f08c23cf640 1 librados: starting objecter 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.241+0000 7f08c23cf640 1 Processor -- start 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.242+0000 7f08c23cf640 1 -- start start 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.242+0000 7f08c23cf640 1 librados: setting wanted keys 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.242+0000 7f08c23cf640 1 librados: calling monclient init 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.242+0000 7f08c23cf640 10 monclient: init 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 10 monclient(hunting): picked mon.noname-b con 0x7f08bc0af8e0 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 10 monclient(hunting): picked mon.noname-a con 0x7f08bc0b0860 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 10 monclient(hunting): picked mon.noname-c con 0x7f08bc1377f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc1d7140 con 0x7f08bc0b0860 2024-03-29T11:40:48.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc133150 con 0x7f08bc0af8e0 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc1d7ce0 con 0x7f08bc1377f0 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.243+0000 7f08c23cf640 10 monclient(hunting): authenticate will time out at 1298.520973s 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.244+0000 7f08bb7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f08bc0b0860 0x7f08bc132330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34166/0 (socket says 172.21.15.22:34166) 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.244+0000 7f08bb7fe640 1 -- 172.21.15.22:0/3025123575 learned_addr learned my addr 172.21.15.22:0/3025123575 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.244+0000 7f08b8ff9640 10 client.?.objecter ms_handle_connect 0x7f08bc0b0860 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.244+0000 7f08b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.245+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048810522 0 0) 0x7f08bc1d7140 con 0x7f08bc0b0860 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.245+0000 7f08b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.245+0000 7f08b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.245+0000 7f08b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.245+0000 7f08b8ff9640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.245+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a8004eb0 con 0x7f08bc0b0860 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.247+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306555522 0 0) 0x7f08a8004eb0 con 0x7f08bc0b0860 2024-03-29T11:40:48.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.247+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08bc1d7140 con 0x7f08bc0b0860 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.247+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08ac001770 con 0x7f08bc0b0860 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.247+0000 7f08b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.247+0000 7f08b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.247+0000 7f08b8ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:48.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.247+0000 7f08b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.247+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2578681984 0 0) 0x7f08bc1d7140 con 0x7f08bc0b0860 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.248+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 >> v1:172.21.15.22:6790/0 conn(0x7f08bc1377f0 legacy=0x7f08bc132a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.248+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 >> v1:172.21.15.204:6789/0 conn(0x7f08bc0af8e0 legacy=0x7f08bc1d75d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.248+0000 7f08b8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.248+0000 7f08b8ff9640 1 monclient: found mon.a 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.248+0000 7f08b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.248+0000 7f08b8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.248+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc135580 con 0x7f08bc0b0860 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.248+0000 7f08b8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08b8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 10 monclient: _renew_subs 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:48.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 1 -- 172.21.15.22:0/3025123575 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc1357d0 con 0x7f08bc0b0860 2024-03-29T11:40:48.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:48.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 10 monclient: _renew_subs 2024-03-29T11:40:48.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:48.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 1 -- 172.21.15.22:0/3025123575 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc1c4600 con 0x7f08bc0b0860 2024-03-29T11:40:48.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.249+0000 7f08c23cf640 1 librados: init done 2024-03-29T11:40:48.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.250+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08ac001d40 con 0x7f08bc0b0860 2024-03-29T11:40:48.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.250+0000 7f08b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:48.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.250+0000 7f08b9ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:48.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.251+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08ac005180 con 0x7f08bc0b0860 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.251+0000 7f08b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.251+0000 7f08b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.251+0000 7f08b8ff9640 10 monclient: dump: 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:48.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.252+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f08ac01b150 con 0x7f08bc0b0860 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.252+0000 7f08bbfff640 10 monclient: get_auth_request con 0x7f08a806dbf0 auth_method 0 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f08ac084c10 con 0x7f08bc0b0860 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08b8ff9640 10 client.4458.objecter ms_dispatch 0x7f08bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08b8ff9640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08b8ff9640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08a27fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08a27fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08a27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08a27fc640 1 -- 172.21.15.22:0/3025123575 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc1d6f70 con 0x7f08bc0b0860 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08b8ff9640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T11:40:48.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.253+0000 7f08b8ff9640 10 client.4458.objecter ms_handle_connect 0x7f08a806dbf0 2024-03-29T11:40:48.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.262+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08ac050930 con 0x7f08bc0b0860 2024-03-29T11:40:48.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.262+0000 7f08b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.263+0000 7f08b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:48.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.613+0000 7f08a27fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.613+0000 7f08a27fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.613+0000 7f08a27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:40:48.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.613+0000 7f08a27fc640 1 -- 172.21.15.22:0/3025123575 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08bc1d72d0 con 0x7f08bc0b0860 2024-03-29T11:40:48.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.614+0000 7f08b8ff9640 1 -- 172.21.15.22:0/3025123575 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f08ac052860 con 0x7f08bc0b0860 2024-03-29T11:40:48.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.614+0000 7f08b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.614+0000 7f08b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:48.619 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:40:48.619 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:48.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.619+0000 7f08a27fc640 10 librados: watch_flush enter 2024-03-29T11:40:48.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.619+0000 7f08a27fc640 10 librados: watch_flush exit 2024-03-29T11:40:48.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.619+0000 7f08a27fc640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T11:40:48.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.619+0000 7f08a27fc640 10 client.4458.objecter successfully canceled tick 2024-03-29T11:40:48.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.619+0000 7f08a27fc640 1 -- 172.21.15.22:0/3025123575 >> v1:172.21.15.204:6816/3848099522 conn(0x7f08a806dbf0 legacy=0x7f08a80700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.619+0000 7f08a27fc640 10 monclient: shutdown 2024-03-29T11:40:48.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.619+0000 7f08a27fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:48.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.619+0000 7f08a27fc640 1 -- 172.21.15.22:0/3025123575 >> v1:172.21.15.22:6789/0 conn(0x7f08bc0b0860 legacy=0x7f08bc132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.620+0000 7f08a27fc640 1 -- 172.21.15.22:0/3025123575 shutdown_connections 2024-03-29T11:40:48.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.620+0000 7f08a27fc640 1 -- 172.21.15.22:0/3025123575 >> 172.21.15.22:0/3025123575 conn(0x7f08bc06b660 msgr2=0x7f08bc13cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:48.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.620+0000 7f08a27fc640 1 -- 172.21.15.22:0/3025123575 shutdown_connections 2024-03-29T11:40:48.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.620+0000 7f08a27fc640 1 -- 172.21.15.22:0/3025123575 wait complete. 2024-03-29T11:40:48.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:48.620+0000 7f08a27fc640 1 librados: shutdown 2024-03-29T11:40:48.639 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:55.446 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:55.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.582+0000 7f9640886640 10 monclient: get_monmap_and_config 2024-03-29T11:40:55.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.582+0000 7f9640886640 10 monclient: build_initial_monmap 2024-03-29T11:40:55.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.582+0000 7f9640886640 10 monclient: monmap: 2024-03-29T11:40:55.583 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:55.583 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:55.584 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:40:55.582314+0000 2024-03-29T11:40:55.584 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:40:55.582314+0000 2024-03-29T11:40:55.584 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:55.584 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:55.584 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:55.584 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:40:55.584 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:40:55.584 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:55.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.582+0000 7f9640886640 1 Processor -- start 2024-03-29T11:40:55.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.583+0000 7f9640886640 1 -- start start 2024-03-29T11:40:55.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.583+0000 7f9640886640 10 monclient: init 2024-03-29T11:40:55.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:55.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:55.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 10 monclient(hunting): picked mon.noname-a con 0x7f963c0b1f60 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:55.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f963a59c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f963c0b1f60 0x7f963c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48020/0 (socket says 172.21.15.22:48020) 2024-03-29T11:40:55.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f963a59c640 1 -- 172.21.15.22:0/3595396860 learned_addr learned my addr 172.21.15.22:0/3595396860 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:55.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 10 monclient(hunting): picked mon.noname-b con 0x7f963c071eb0 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:55.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 10 monclient(hunting): picked mon.noname-c con 0x7f963c143d10 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 1 -- 172.21.15.22:0/3595396860 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06ef80 con 0x7f963c0b1f60 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 1 -- 172.21.15.22:0/3595396860 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06f260 con 0x7f963c143d10 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 1 -- 172.21.15.22:0/3595396860 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c0b40c0 con 0x7f963c071eb0 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.584+0000 7f9640886640 10 monclient(hunting): authenticate will time out at 1305.860882s 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.585+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140421283 0 0) 0x7f963c0b40c0 con 0x7f963c071eb0 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.585+0000 7f963959a640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.585+0000 7f963959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.585+0000 7f963959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.585+0000 7f963959a640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.585+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9624003520 con 0x7f963c071eb0 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.585+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4136189601 0 0) 0x7f9624003520 con 0x7f963c071eb0 2024-03-29T11:40:55.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.585+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 >> v1:172.21.15.204:6789/0 conn(0x7f963c143d10 legacy=0x7f963c146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:55.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 >> v1:172.21.15.22:6789/0 conn(0x7f963c0b1f60 legacy=0x7f963c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 10 monclient: _finish_hunting 0 2024-03-29T11:40:55.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 1 monclient: found mon.noname-b 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c153d50 con 0x7f963c071eb0 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 10 monclient: _finish_auth 0 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 10 monclient: _check_auth_tickets 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9630001690 con 0x7f963c071eb0 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.587+0000 7f963959a640 10 monclient: dump: 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.593 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f9640886640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f9640886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9630001860 con 0x7f963c071eb0 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f963959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f963959a640 1 -- 172.21.15.22:0/3595396860 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9630001cb0 con 0x7f963c071eb0 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f963959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f963959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:55.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f963959a640 10 monclient: dump: 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f9640886640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.588+0000 7f9640886640 4 set_mon_vals no callback set 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.589+0000 7f9640886640 10 monclient: shutdown 2024-03-29T11:40:55.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.589+0000 7f9640886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.589+0000 7f9640886640 1 -- 172.21.15.22:0/3595396860 >> v1:172.21.15.22:6790/0 conn(0x7f963c071eb0 legacy=0x7f963c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.589+0000 7f9640886640 1 -- 172.21.15.22:0/3595396860 shutdown_connections 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.589+0000 7f9640886640 1 -- 172.21.15.22:0/3595396860 >> 172.21.15.22:0/3595396860 conn(0x7f963c066a00 msgr2=0x7f963c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.590+0000 7f9640886640 1 -- 172.21.15.22:0/3595396860 shutdown_connections 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.590+0000 7f9640886640 1 -- 172.21.15.22:0/3595396860 wait complete. 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.590+0000 7f9640886640 10 monclient: build_initial_monmap 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.590+0000 7f9640886640 10 monclient: monmap: 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:55.596 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.591+0000 7f9640886640 1 librados: starting msgr at 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.591+0000 7f9640886640 1 librados: starting objecter 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.591+0000 7f9640886640 1 Processor -- start 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.591+0000 7f9640886640 1 -- start start 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.591+0000 7f9640886640 1 librados: setting wanted keys 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.591+0000 7f9640886640 1 librados: calling monclient init 2024-03-29T11:40:55.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.591+0000 7f9640886640 10 monclient: init 2024-03-29T11:40:55.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:55.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:55.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 10 monclient(hunting): picked mon.noname-b con 0x7f963c0b1f60 addr v1:172.21.15.204:6789/0 2024-03-29T11:40:55.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 10 monclient(hunting): picked mon.noname-a con 0x7f963c143d10 addr v1:172.21.15.22:6789/0 2024-03-29T11:40:55.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 10 monclient(hunting): picked mon.noname-c con 0x7f963c141090 addr v1:172.21.15.22:6790/0 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06f8e0 con 0x7f963c143d10 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06ff70 con 0x7f963c0b1f60 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c136970 con 0x7f963c141090 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f9640886640 10 monclient(hunting): authenticate will time out at 1305.868319s 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f963ad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f963c141090 0x7f963c1e5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:36644/0 (socket says 172.21.15.22:36644) 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f963ad9d640 1 -- 172.21.15.22:0/2664675592 learned_addr learned my addr 172.21.15.22:0/2664675592 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 10 client.?.objecter ms_handle_connect 0x7f963c141090 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 10 client.?.objecter ms_handle_connect 0x7f963c143d10 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:55.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368841848 0 0) 0x7f963c06f8e0 con 0x7f963c143d10 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.592+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f961c004eb0 con 0x7f963c143d10 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935118137 0 0) 0x7f963c136970 con 0x7f963c141090 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c06f8e0 con 0x7f963c141090 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609544031 0 0) 0x7f961c004eb0 con 0x7f963c143d10 2024-03-29T11:40:55.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f963c136970 con 0x7f963c143d10 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9630001b70 con 0x7f963c143d10 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 10 monclient(hunting): dump: 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:55.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652110707 0 0) 0x7f963c06f8e0 con 0x7f963c141090 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.593+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f961c004eb0 con 0x7f963c141090 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.594+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9634001a90 con 0x7f963c141090 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.594+0000 7f962affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.594+0000 7f962affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.594+0000 7f962affd640 10 monclient(hunting): dump: 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:55.606 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.594+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 159144532 0 0) 0x7f961c004eb0 con 0x7f963c141090 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.594+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 >> v1:172.21.15.204:6789/0 conn(0x7f963c0b1f60 legacy=0x7f963c070430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 >> v1:172.21.15.22:6789/0 conn(0x7f963c143d10 legacy=0x7f963c070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 1 monclient: found mon.c 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c136b00 con 0x7f963c141090 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 10 monclient: _finish_auth 0 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:55.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f962affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f9640886640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f9640886640 10 monclient: _renew_subs 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f9640886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.595+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c1420f0 con 0x7f963c141090 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f9640886640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f9640886640 10 monclient: _renew_subs 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f9640886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c1e75c0 con 0x7f963c141090 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f9640886640 1 librados: init done 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9634001fa0 con 0x7f963c141090 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f962affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9634004d80 con 0x7f963c141090 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f962affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f962affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:55.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f962affd640 10 monclient: dump: 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.596+0000 7f9638d99640 4 set_mon_vals no callback set 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f963401ad50 con 0x7f963c141090 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f9640886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f9640886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f9640886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963c070150 con 0x7f963c141090 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f9634084cb0 con 0x7f963c141090 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f962affd640 10 client.4472.objecter ms_dispatch 0x7f963c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f962affd640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f962affd640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f962affd640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.598+0000 7f963a59c640 10 monclient: get_auth_request con 0x7f961c06ded0 auth_method 0 2024-03-29T11:40:55.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.599+0000 7f962affd640 10 client.4472.objecter ms_handle_connect 0x7f961c06ded0 2024-03-29T11:40:55.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.606+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f963c070150 con 0x7f963c141090 2024-03-29T11:40:55.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.606+0000 7f962affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.606+0000 7f962affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:55.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.932+0000 7f9640886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:55.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.932+0000 7f9640886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:55.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.932+0000 7f9640886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:40:55.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.932+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f963c070100 con 0x7f963c141090 2024-03-29T11:40:55.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.934+0000 7f962affd640 1 -- 172.21.15.22:0/2664675592 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f96340503c0 con 0x7f963c141090 2024-03-29T11:40:55.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.934+0000 7f962affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:55.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.934+0000 7f962affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:55.936 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:40:55.936 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:55.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.937+0000 7f9640886640 10 librados: watch_flush enter 2024-03-29T11:40:55.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.937+0000 7f9640886640 10 librados: watch_flush exit 2024-03-29T11:40:55.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.937+0000 7f9640886640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T11:40:55.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.937+0000 7f9640886640 10 client.4472.objecter successfully canceled tick 2024-03-29T11:40:55.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 >> v1:172.21.15.204:6816/3848099522 conn(0x7f961c06ded0 legacy=0x7f961c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 10 monclient: shutdown 2024-03-29T11:40:55.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:55.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 >> v1:172.21.15.22:6790/0 conn(0x7f963c141090 legacy=0x7f963c1e5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 shutdown_connections 2024-03-29T11:40:55.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 >> 172.21.15.22:0/2664675592 conn(0x7f963c066a00 msgr2=0x7f963c140630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:55.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 shutdown_connections 2024-03-29T11:40:55.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 1 -- 172.21.15.22:0/2664675592 wait complete. 2024-03-29T11:40:55.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:40:55.938+0000 7f9640886640 1 librados: shutdown 2024-03-29T11:40:55.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:02.763 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:02.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.915+0000 7f14d4414640 10 monclient: get_monmap_and_config 2024-03-29T11:41:02.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.916+0000 7f14d4414640 10 monclient: build_initial_monmap 2024-03-29T11:41:02.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.916+0000 7f14d4414640 10 monclient: monmap: 2024-03-29T11:41:02.917 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:02.917 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:41:02.916324+0000 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:41:02.916324+0000 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:41:02.918 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:02.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.917+0000 7f14d4414640 1 Processor -- start 2024-03-29T11:41:02.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.917+0000 7f14d4414640 1 -- start start 2024-03-29T11:41:02.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.917+0000 7f14d4414640 10 monclient: init 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 10 monclient(hunting): picked mon.noname-b con 0x7f14cc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 10 monclient(hunting): picked mon.noname-a con 0x7f14cc13f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 10 monclient(hunting): picked mon.noname-c con 0x7f14cc0a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc0aaef0 con 0x7f14cc13f650 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc145bd0 con 0x7f14cc0a9d70 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc145eb0 con 0x7f14cc1457f0 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d4414640 10 monclient(hunting): authenticate will time out at 1313.194443s 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d29b1640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f14cc0a9d70 0x7f14cc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:47252/0 (socket says 172.21.15.22:47252) 2024-03-29T11:41:02.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.918+0000 7f14d29b1640 1 -- 172.21.15.22:0/1573111111 learned_addr learned my addr 172.21.15.22:0/1573111111 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548431281 0 0) 0x7f14cc0aaef0 con 0x7f14cc13f650 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14b8003520 con 0x7f14cc13f650 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514879226 0 0) 0x7f14cc145bd0 con 0x7f14cc0a9d70 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14cc0aaef0 con 0x7f14cc0a9d70 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1033271060 0 0) 0x7f14b8003520 con 0x7f14cc13f650 2024-03-29T11:41:02.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 >> v1:172.21.15.22:6790/0 conn(0x7f14cc1457f0 legacy=0x7f14cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 >> v1:172.21.15.204:6789/0 conn(0x7f14cc0a9d70 legacy=0x7f14cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient: _finish_hunting 0 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 monclient: found mon.noname-a 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc146340 con 0x7f14cc13f650 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient: _finish_auth 0 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient: _check_auth_tickets 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d4414640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d4414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f14c0001690 con 0x7f14cc13f650 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d11ae640 10 monclient: dump: 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:02.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.919+0000 7f14d4414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.920+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14c00017f0 con 0x7f14cc13f650 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.920+0000 7f14d11ae640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.920+0000 7f14d11ae640 1 -- 172.21.15.22:0/1573111111 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f14c0001c00 con 0x7f14cc13f650 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.920+0000 7f14d11ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.920+0000 7f14d11ae640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.920+0000 7f14d11ae640 10 monclient: dump: 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.926 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.920+0000 7f14d4414640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.920+0000 7f14d4414640 4 set_mon_vals no callback set 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.921+0000 7f14d4414640 10 monclient: shutdown 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.921+0000 7f14d4414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.921+0000 7f14d4414640 1 -- 172.21.15.22:0/1573111111 >> v1:172.21.15.22:6789/0 conn(0x7f14cc13f650 legacy=0x7f14cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.921+0000 7f14d4414640 1 -- 172.21.15.22:0/1573111111 shutdown_connections 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.921+0000 7f14d4414640 1 -- 172.21.15.22:0/1573111111 >> 172.21.15.22:0/1573111111 conn(0x7f14cc066a00 msgr2=0x7f14cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.921+0000 7f14d4414640 1 -- 172.21.15.22:0/1573111111 shutdown_connections 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.921+0000 7f14d4414640 1 -- 172.21.15.22:0/1573111111 wait complete. 2024-03-29T11:41:02.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.922+0000 7f14d4414640 10 monclient: build_initial_monmap 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.922+0000 7f14d4414640 10 monclient: monmap: 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.923+0000 7f14d4414640 1 librados: starting msgr at 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.923+0000 7f14d4414640 1 librados: starting objecter 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.923+0000 7f14d4414640 1 Processor -- start 2024-03-29T11:41:02.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.923+0000 7f14d4414640 1 -- start start 2024-03-29T11:41:02.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.923+0000 7f14d4414640 1 librados: setting wanted keys 2024-03-29T11:41:02.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.923+0000 7f14d4414640 1 librados: calling monclient init 2024-03-29T11:41:02.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.923+0000 7f14d4414640 10 monclient: init 2024-03-29T11:41:02.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:02.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 10 monclient(hunting): picked mon.noname-a con 0x7f14cc0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 10 monclient(hunting): picked mon.noname-c con 0x7f14cc13f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 10 monclient(hunting): picked mon.noname-b con 0x7f14cc1457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc06f460 con 0x7f14cc0a9d70 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc06f6b0 con 0x7f14cc1457f0 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc1dbd30 con 0x7f14cc13f650 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d4414640 10 monclient(hunting): authenticate will time out at 1313.201632s 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d19af640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f14cc13f650 0x7f14cc1dc250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60176/0 (socket says 172.21.15.22:60176) 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14d19af640 1 -- 172.21.15.22:0/1307658012 learned_addr learned my addr 172.21.15.22:0/1307658012 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14beffd640 10 client.?.objecter ms_handle_connect 0x7f14cc13f650 2024-03-29T11:41:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.925+0000 7f14beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 client.?.objecter ms_handle_connect 0x7f14cc0a9d70 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046864908 0 0) 0x7f14cc1dbd30 con 0x7f14cc13f650 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14b0004eb0 con 0x7f14cc13f650 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878978300 0 0) 0x7f14cc06f460 con 0x7f14cc0a9d70 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14cc1dbd30 con 0x7f14cc0a9d70 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 client.?.objecter ms_handle_connect 0x7f14cc1457f0 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.926+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708559091 0 0) 0x7f14cc06f6b0 con 0x7f14cc1457f0 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14cc06f460 con 0x7f14cc1457f0 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3221257448 0 0) 0x7f14cc1dbd30 con 0x7f14cc0a9d70 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14cc06f6b0 con 0x7f14cc0a9d70 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f14c8001800 con 0x7f14cc0a9d70 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): dump: 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4066983534 0 0) 0x7f14cc06f460 con 0x7f14cc1457f0 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14cc1dbd30 con 0x7f14cc1457f0 2024-03-29T11:41:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f14c4001cb0 con 0x7f14cc1457f0 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): dump: 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:02.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1519076691 0 0) 0x7f14b0004eb0 con 0x7f14cc13f650 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14cc06f460 con 0x7f14cc13f650 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f14c0001690 con 0x7f14cc13f650 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.927+0000 7f14beffd640 10 monclient(hunting): dump: 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:02.939 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1173636297 0 0) 0x7f14cc1dbd30 con 0x7f14cc1457f0 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 >> v1:172.21.15.22:6790/0 conn(0x7f14cc13f650 legacy=0x7f14cc1dc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 >> v1:172.21.15.22:6789/0 conn(0x7f14cc0a9d70 legacy=0x7f14cc13afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 1 monclient: found mon.b 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc1dc960 con 0x7f14cc1457f0 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:41:02.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 10 monclient: _renew_subs 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc138cc0 con 0x7f14cc1457f0 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 10 monclient: _renew_subs 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc139010 con 0x7f14cc1457f0 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d4414640 1 librados: init done 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14c40020c0 con 0x7f14cc1457f0 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.928+0000 7f14d09ad640 4 set_mon_vals no callback set 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.929+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f14c4004d00 con 0x7f14cc1457f0 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.929+0000 7f14beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.929+0000 7f14beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.929+0000 7f14beffd640 10 monclient: dump: 2024-03-29T11:41:02.941 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.930+0000 7f14d4414640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.931+0000 7f14d4414640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.931+0000 7f14d4414640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.931+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b4005080 con 0x7f14cc1457f0 2024-03-29T11:41:02.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.934+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f14c4004e60 con 0x7f14cc1457f0 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.935+0000 7f14d21b0640 10 monclient: get_auth_request con 0x7f14b006e120 auth_method 0 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.935+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f14c4084bf0 con 0x7f14cc1457f0 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.935+0000 7f14beffd640 10 client.4432.objecter ms_dispatch 0x7f14cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.935+0000 7f14beffd640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.935+0000 7f14beffd640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.935+0000 7f14beffd640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.935+0000 7f14beffd640 10 client.4432.objecter ms_handle_connect 0x7f14b006e120 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.941+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14c4050300 con 0x7f14cc1457f0 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.941+0000 7f14beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:02.941+0000 7f14beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:03.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.321+0000 7f14d4414640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.321+0000 7f14d4414640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.321+0000 7f14d4414640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:03.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.321+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14b4005330 con 0x7f14cc1457f0 2024-03-29T11:41:03.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.322+0000 7f14beffd640 1 -- 172.21.15.22:0/1307658012 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f14c40012e0 con 0x7f14cc1457f0 2024-03-29T11:41:03.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.322+0000 7f14beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.322+0000 7f14beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:03.324 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:41:03.324 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 10 librados: watch_flush enter 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 10 librados: watch_flush exit 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 10 client.4432.objecter successfully canceled tick 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 >> v1:172.21.15.204:6816/3848099522 conn(0x7f14b006e120 legacy=0x7f14b00705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 10 monclient: shutdown 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 >> v1:172.21.15.204:6789/0 conn(0x7f14cc1457f0 legacy=0x7f14cc1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 shutdown_connections 2024-03-29T11:41:03.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.326+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 >> 172.21.15.22:0/1307658012 conn(0x7f14cc066a00 msgr2=0x7f14cc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:03.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.327+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 shutdown_connections 2024-03-29T11:41:03.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.327+0000 7f14d4414640 1 -- 172.21.15.22:0/1307658012 wait complete. 2024-03-29T11:41:03.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:03.327+0000 7f14d4414640 1 librados: shutdown 2024-03-29T11:41:03.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:10.150 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.300+0000 7f5ebffff640 10 monclient: get_monmap_and_config 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.300+0000 7f5ebffff640 10 monclient: build_initial_monmap 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.300+0000 7f5ebffff640 10 monclient: monmap: 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:41:10.301838+0000 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:41:10.301838+0000 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:10.303 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:10.304 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:41:10.304 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:41:10.304 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.301+0000 7f5ebffff640 1 Processor -- start 2024-03-29T11:41:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.301+0000 7f5ebffff640 1 -- start start 2024-03-29T11:41:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.301+0000 7f5ebffff640 10 monclient: init 2024-03-29T11:41:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:10.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 10 monclient(hunting): picked mon.noname-c con 0x7f5ec00b2800 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5ec00b3220 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 10 monclient(hunting): picked mon.noname-b con 0x7f5ec01439c0 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec00b3840 con 0x7f5ec00b3220 2024-03-29T11:41:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec006f230 con 0x7f5ec00b2800 2024-03-29T11:41:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec006ef50 con 0x7f5ec01439c0 2024-03-29T11:41:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:10.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.302+0000 7f5ebffff640 10 monclient(hunting): authenticate will time out at 1320.579384s 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.304+0000 7f5ebf7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5ec01439c0 0x7f5ec0151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55350/0 (socket says 172.21.15.22:55350) 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.304+0000 7f5ebf7fe640 1 -- 172.21.15.22:0/1539598192 learned_addr learned my addr 172.21.15.22:0/1539598192 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.305+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753225232 0 0) 0x7f5ec006ef50 con 0x7f5ec01439c0 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.305+0000 7f5ebdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.305+0000 7f5ebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.305+0000 7f5ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.305+0000 7f5ebdffb640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.305+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea8003520 con 0x7f5ec01439c0 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.305+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3853720459 0 0) 0x7f5ea8003520 con 0x7f5ec01439c0 2024-03-29T11:41:10.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.305+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 >> v1:172.21.15.204:6789/0 conn(0x7f5ec00b2800 legacy=0x7f5ec01432b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:10.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.306+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 >> v1:172.21.15.22:6789/0 conn(0x7f5ec00b3220 legacy=0x7f5ec0146710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:10.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:10.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 1 monclient: found mon.noname-b 2024-03-29T11:41:10.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec0071d30 con 0x7f5ec01439c0 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5eb8001a10 con 0x7f5ec01439c0 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebdffb640 10 monclient: dump: 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:10.312 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebffff640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.307+0000 7f5ebffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.308+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb8001be0 con 0x7f5ec01439c0 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.308+0000 7f5ebdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.308+0000 7f5ebdffb640 1 -- 172.21.15.22:0/1539598192 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5eb8002030 con 0x7f5ec01439c0 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.308+0000 7f5ebdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.309+0000 7f5ebdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.309+0000 7f5ebdffb640 10 monclient: dump: 2024-03-29T11:41:10.313 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:10.314 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:10.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.309+0000 7f5ebffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:10.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.309+0000 7f5ebffff640 4 set_mon_vals no callback set 2024-03-29T11:41:10.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.310+0000 7f5ebffff640 10 monclient: shutdown 2024-03-29T11:41:10.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.310+0000 7f5ebffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.310+0000 7f5ebffff640 1 -- 172.21.15.22:0/1539598192 >> v1:172.21.15.22:6790/0 conn(0x7f5ec01439c0 legacy=0x7f5ec0151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.310+0000 7f5ebffff640 1 -- 172.21.15.22:0/1539598192 shutdown_connections 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.310+0000 7f5ebffff640 1 -- 172.21.15.22:0/1539598192 >> 172.21.15.22:0/1539598192 conn(0x7f5ec0066a00 msgr2=0x7f5ec0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.310+0000 7f5ebffff640 1 -- 172.21.15.22:0/1539598192 shutdown_connections 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.310+0000 7f5ebffff640 1 -- 172.21.15.22:0/1539598192 wait complete. 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.311+0000 7f5ebffff640 10 monclient: build_initial_monmap 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.311+0000 7f5ebffff640 10 monclient: monmap: 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:10.317 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.311+0000 7f5ebffff640 1 librados: starting msgr at 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.311+0000 7f5ebffff640 1 librados: starting objecter 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.311+0000 7f5ebffff640 1 Processor -- start 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.312+0000 7f5ebffff640 1 -- start start 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.312+0000 7f5ebffff640 1 librados: setting wanted keys 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.312+0000 7f5ebffff640 1 librados: calling monclient init 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.312+0000 7f5ebffff640 10 monclient: init 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:10.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 10 monclient(hunting): picked mon.noname-b con 0x7f5ec00b2800 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5ec00b3220 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 10 monclient(hunting): picked mon.noname-c con 0x7f5ec01439c0 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec0149d70 con 0x7f5ec00b3220 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec0120f20 con 0x7f5ec00b2800 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec0149a00 con 0x7f5ec01439c0 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.313+0000 7f5ebffff640 10 monclient(hunting): authenticate will time out at 1320.590264s 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ebf7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5ec01439c0 0x7f5ec01db790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57692/0 (socket says 172.21.15.22:57692) 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ebf7fe640 1 -- 172.21.15.22:0/1251920262 learned_addr learned my addr 172.21.15.22:0/1251920262 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 client.?.objecter ms_handle_connect 0x7f5ec01439c0 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882581527 0 0) 0x7f5ec0149a00 con 0x7f5ec01439c0 2024-03-29T11:41:10.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:41:10.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eac004eb0 con 0x7f5ec01439c0 2024-03-29T11:41:10.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3826750514 0 0) 0x7f5eac004eb0 con 0x7f5ec01439c0 2024-03-29T11:41:10.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ec0149a00 con 0x7f5ec01439c0 2024-03-29T11:41:10.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5eb8001f30 con 0x7f5ec01439c0 2024-03-29T11:41:10.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:10.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 monclient(hunting): dump: 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.314+0000 7f5ea7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.315+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 609305911 0 0) 0x7f5ec0149a00 con 0x7f5ec01439c0 2024-03-29T11:41:10.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.315+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 >> v1:172.21.15.204:6789/0 conn(0x7f5ec00b2800 legacy=0x7f5ec014a1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.316+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 >> v1:172.21.15.22:6789/0 conn(0x7f5ec00b3220 legacy=0x7f5ec01db080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.316+0000 7f5ea7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.316+0000 7f5ea7fff640 1 monclient: found mon.c 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ea7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec011f9e0 con 0x7f5ec01439c0 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ea7fff640 10 monclient: _finish_auth 0 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ea7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ea7fff640 10 client.?.objecter ms_handle_connect 0x7f5ec00b3220 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ea7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 10 monclient: _renew_subs 2024-03-29T11:41:10.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec011fc30 con 0x7f5ec01439c0 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 10 monclient: _renew_subs 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec01e6d70 con 0x7f5ec01439c0 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.317+0000 7f5ebffff640 1 librados: init done 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.318+0000 7f5ebffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.318+0000 7f5ebffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.318+0000 7f5ebffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:10.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.318+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec004f730 con 0x7f5ec01439c0 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.319+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb8002090 con 0x7f5ec01439c0 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.319+0000 7f5ea7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.319+0000 7f5ebd7fa640 4 set_mon_vals no callback set 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.319+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5eb8005030 con 0x7f5ec01439c0 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.319+0000 7f5ea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.319+0000 7f5ea7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.320+0000 7f5ea7fff640 10 monclient: dump: 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:10.326 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.321+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f5eb801b000 con 0x7f5ec01439c0 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.322+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f5eb8084e70 con 0x7f5ec01439c0 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.322+0000 7f5ea7fff640 10 client.4484.objecter ms_dispatch 0x7f5ec0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.322+0000 7f5ebeffd640 10 monclient: get_auth_request con 0x7f5eac06dca0 auth_method 0 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.322+0000 7f5ea7fff640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.322+0000 7f5ea7fff640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:10.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.322+0000 7f5ea7fff640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T11:41:10.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.322+0000 7f5ea7fff640 10 client.4484.objecter ms_handle_connect 0x7f5eac06dca0 2024-03-29T11:41:10.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.329+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5eb8050580 con 0x7f5ec01439c0 2024-03-29T11:41:10.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.329+0000 7f5ea7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.329+0000 7f5ea7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:10.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.655+0000 7f5ebffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.655+0000 7f5ebffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.655+0000 7f5ebffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:10.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.655+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ec0145920 con 0x7f5ec01439c0 2024-03-29T11:41:10.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.657+0000 7f5ea7fff640 1 -- 172.21.15.22:0/1251920262 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f5eb8052680 con 0x7f5ec01439c0 2024-03-29T11:41:10.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.657+0000 7f5ea7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.657+0000 7f5ea7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:10.660 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:41:10.660 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:10.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.660+0000 7f5ebffff640 10 librados: watch_flush enter 2024-03-29T11:41:10.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.660+0000 7f5ebffff640 10 librados: watch_flush exit 2024-03-29T11:41:10.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.660+0000 7f5ebffff640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T11:41:10.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.660+0000 7f5ebffff640 10 client.4484.objecter successfully canceled tick 2024-03-29T11:41:10.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.660+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 >> v1:172.21.15.204:6816/3848099522 conn(0x7f5eac06dca0 legacy=0x7f5eac070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.660+0000 7f5ebffff640 10 monclient: shutdown 2024-03-29T11:41:10.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.660+0000 7f5ebffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:10.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.660+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 >> v1:172.21.15.22:6790/0 conn(0x7f5ec01439c0 legacy=0x7f5ec01db790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.662+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 shutdown_connections 2024-03-29T11:41:10.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.662+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 >> 172.21.15.22:0/1251920262 conn(0x7f5ec0066a00 msgr2=0x7f5ec0141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:10.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.663+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 shutdown_connections 2024-03-29T11:41:10.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.663+0000 7f5ebffff640 1 -- 172.21.15.22:0/1251920262 wait complete. 2024-03-29T11:41:10.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:10.663+0000 7f5ebffff640 1 librados: shutdown 2024-03-29T11:41:10.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:17.489 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:17.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.621+0000 7fb64f325640 10 monclient: get_monmap_and_config 2024-03-29T11:41:17.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.621+0000 7fb64f325640 10 monclient: build_initial_monmap 2024-03-29T11:41:17.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.621+0000 7fb64f325640 10 monclient: monmap: 2024-03-29T11:41:17.624 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:17.624 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:17.624 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:41:17.622917+0000 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:41:17.622917+0000 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:41:17.625 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.622+0000 7fb64f325640 1 Processor -- start 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.623+0000 7fb64f325640 1 -- start start 2024-03-29T11:41:17.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.623+0000 7fb64f325640 10 monclient: init 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 10 monclient(hunting): picked mon.noname-c con 0x7fb6481457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 10 monclient(hunting): picked mon.noname-a con 0x7fb64813f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 10 monclient(hunting): picked mon.noname-b con 0x7fb6480a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6480aaef0 con 0x7fb64813f650 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb648145bd0 con 0x7fb6481457f0 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb648145eb0 con 0x7fb6480a9d70 2024-03-29T11:41:17.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64f325640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64d8c2640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb6480a9d70 0x7fb64814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57422/0 (socket says 172.21.15.22:57422) 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.624+0000 7fb64d8c2640 1 -- 172.21.15.22:0/3956610054 learned_addr learned my addr 172.21.15.22:0/3956610054 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb64f325640 10 monclient(hunting): authenticate will time out at 1327.902050s 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067564453 0 0) 0x7fb648145eb0 con 0x7fb6480a9d70 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb634003520 con 0x7fb6480a9d70 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2230569321 0 0) 0x7fb634003520 con 0x7fb6480a9d70 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 >> v1:172.21.15.204:6789/0 conn(0x7fb6481457f0 legacy=0x7fb64813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.625+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 >> v1:172.21.15.22:6789/0 conn(0x7fb64813f650 legacy=0x7fb6480b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:17.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.626+0000 7fb63ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.626+0000 7fb63ffff640 1 monclient: found mon.noname-b 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.626+0000 7fb63ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb648146340 con 0x7fb6480a9d70 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 10 monclient: _finish_auth 0 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:17.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb640001a10 con 0x7fb6480a9d70 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 10 monclient: dump: 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:17.634 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb64f325640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb64f325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb640001c30 con 0x7fb6480a9d70 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.627+0000 7fb63ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.628+0000 7fb64f325640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.628+0000 7fb64f325640 4 set_mon_vals no callback set 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.628+0000 7fb63ffff640 1 -- 172.21.15.22:0/3956610054 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb640002040 con 0x7fb6480a9d70 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.628+0000 7fb63ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.628+0000 7fb63ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.628+0000 7fb63ffff640 10 monclient: dump: 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:17.635 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.629+0000 7fb64f325640 10 monclient: shutdown 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.629+0000 7fb64f325640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.629+0000 7fb64f325640 1 -- 172.21.15.22:0/3956610054 >> v1:172.21.15.22:6790/0 conn(0x7fb6480a9d70 legacy=0x7fb64814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.629+0000 7fb64f325640 1 -- 172.21.15.22:0/3956610054 shutdown_connections 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.629+0000 7fb64f325640 1 -- 172.21.15.22:0/3956610054 >> 172.21.15.22:0/3956610054 conn(0x7fb648066a00 msgr2=0x7fb648066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:17.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.629+0000 7fb64f325640 1 -- 172.21.15.22:0/3956610054 shutdown_connections 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.629+0000 7fb64f325640 1 -- 172.21.15.22:0/3956610054 wait complete. 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.630+0000 7fb64f325640 10 monclient: build_initial_monmap 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.630+0000 7fb64f325640 10 monclient: monmap: 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:41:17.637 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:17.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.630+0000 7fb64f325640 1 librados: starting msgr at 2024-03-29T11:41:17.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.630+0000 7fb64f325640 1 librados: starting objecter 2024-03-29T11:41:17.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 1 Processor -- start 2024-03-29T11:41:17.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 1 -- start start 2024-03-29T11:41:17.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 1 librados: setting wanted keys 2024-03-29T11:41:17.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 1 librados: calling monclient init 2024-03-29T11:41:17.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 10 monclient: init 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 10 monclient(hunting): picked mon.noname-c con 0x7fb6480a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 10 monclient(hunting): picked mon.noname-a con 0x7fb64813f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 10 monclient(hunting): picked mon.noname-b con 0x7fb6481457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6481dbe40 con 0x7fb64813f650 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb648141fd0 con 0x7fb6481457f0 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64813fed0 con 0x7fb6480a9d70 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.631+0000 7fb64f325640 10 monclient(hunting): authenticate will time out at 1327.908970s 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.632+0000 7fb64d0c1640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb6480a9d70 0x7fb6481350e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59628/0 (socket says 172.21.15.22:59628) 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.632+0000 7fb64d0c1640 1 -- 172.21.15.22:0/2831882365 learned_addr learned my addr 172.21.15.22:0/2831882365 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 10 client.?.objecter ms_handle_connect 0x7fb6480a9d70 2024-03-29T11:41:17.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273008836 0 0) 0x7fb64813fed0 con 0x7fb6480a9d70 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c004eb0 con 0x7fb6480a9d70 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 10 client.?.objecter ms_handle_connect 0x7fb6481457f0 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.633+0000 7fb63dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3959533800 0 0) 0x7fb62c004eb0 con 0x7fb6480a9d70 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb64813fed0 con 0x7fb6480a9d70 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918248026 0 0) 0x7fb648141fd0 con 0x7fb6481457f0 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c005590 con 0x7fb6481457f0 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb644001930 con 0x7fb6480a9d70 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 10 monclient(hunting): dump: 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:17.649 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.634+0000 7fb63dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 10 client.?.objecter ms_handle_connect 0x7fb64813f650 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 858837708 0 0) 0x7fb64813fed0 con 0x7fb6480a9d70 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 >> v1:172.21.15.204:6789/0 conn(0x7fb6481457f0 legacy=0x7fb648142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 >> v1:172.21.15.22:6789/0 conn(0x7fb64813f650 legacy=0x7fb6481db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 1 monclient: found mon.c 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:17.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb648142b70 con 0x7fb6480a9d70 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb63dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 10 monclient: _renew_subs 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb648142dc0 con 0x7fb6480a9d70 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 10 monclient: _renew_subs 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64813b8e0 con 0x7fb6480a9d70 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.635+0000 7fb64f325640 1 librados: init done 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.636+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6440035f0 con 0x7fb6480a9d70 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.636+0000 7fb63dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:17.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.636+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb644001930 con 0x7fb6480a9d70 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.636+0000 7fb63dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.636+0000 7fb63dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.636+0000 7fb63dffb640 10 monclient: dump: 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:17.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.636+0000 7fb63f7fe640 4 set_mon_vals no callback set 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.636+0000 7fb64f325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.637+0000 7fb64f325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.637+0000 7fb64f325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.637+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb628005080 con 0x7fb6480a9d70 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.639+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fb644003860 con 0x7fb6480a9d70 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.640+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fb644084870 con 0x7fb6480a9d70 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.640+0000 7fb63dffb640 10 client.4490.objecter ms_dispatch 0x7fb648000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.640+0000 7fb63dffb640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.640+0000 7fb63dffb640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.640+0000 7fb63dffb640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.640+0000 7fb64c8c0640 10 monclient: get_auth_request con 0x7fb62c06e1d0 auth_method 0 2024-03-29T11:41:17.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.640+0000 7fb63dffb640 10 client.4490.objecter ms_handle_connect 0x7fb62c06e1d0 2024-03-29T11:41:17.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.647+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb64404ff80 con 0x7fb6480a9d70 2024-03-29T11:41:17.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.647+0000 7fb63dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.647+0000 7fb63dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:17.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.958+0000 7fb64f325640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:17.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.958+0000 7fb64f325640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:17.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.958+0000 7fb64f325640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:17.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.958+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb628005330 con 0x7fb6480a9d70 2024-03-29T11:41:17.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.958+0000 7fb63dffb640 1 -- 172.21.15.22:0/2831882365 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fb644052080 con 0x7fb6480a9d70 2024-03-29T11:41:17.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.958+0000 7fb63dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:17.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.958+0000 7fb63dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:17.961 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:41:17.962 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:17.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.961+0000 7fb64f325640 10 librados: watch_flush enter 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.961+0000 7fb64f325640 10 librados: watch_flush exit 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.961+0000 7fb64f325640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.961+0000 7fb64f325640 10 client.4490.objecter successfully canceled tick 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 >> v1:172.21.15.204:6816/3848099522 conn(0x7fb62c06e1d0 legacy=0x7fb62c070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 10 monclient: shutdown 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 >> v1:172.21.15.22:6790/0 conn(0x7fb6480a9d70 legacy=0x7fb6481350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 shutdown_connections 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 >> 172.21.15.22:0/2831882365 conn(0x7fb648066a00 msgr2=0x7fb648066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 shutdown_connections 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 1 -- 172.21.15.22:0/2831882365 wait complete. 2024-03-29T11:41:17.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:17.962+0000 7fb64f325640 1 librados: shutdown 2024-03-29T11:41:17.978 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:24.785 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.938+0000 7f08c810e640 10 monclient: get_monmap_and_config 2024-03-29T11:41:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.938+0000 7f08c810e640 10 monclient: build_initial_monmap 2024-03-29T11:41:24.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.938+0000 7f08c810e640 10 monclient: monmap: 2024-03-29T11:41:24.942 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:24.942 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:24.942 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:41:24.939781+0000 2024-03-29T11:41:24.942 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:41:24.939781+0000 2024-03-29T11:41:24.942 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:24.942 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:24.942 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:24.943 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:41:24.943 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:41:24.943 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:24.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.940+0000 7f08c810e640 1 Processor -- start 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.940+0000 7f08c810e640 1 -- start start 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.940+0000 7f08c810e640 10 monclient: init 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 10 monclient(hunting): picked mon.noname-c con 0x7f08c01457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 10 monclient(hunting): picked mon.noname-a con 0x7f08c013f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 10 monclient(hunting): picked mon.noname-b con 0x7f08c00a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c00aaef0 con 0x7f08c013f650 2024-03-29T11:41:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c0145bd0 con 0x7f08c01457f0 2024-03-29T11:41:24.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c0145eb0 con 0x7f08c00a9d70 2024-03-29T11:41:24.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:24.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.941+0000 7f08c810e640 10 monclient(hunting): authenticate will time out at 1335.218526s 2024-03-29T11:41:24.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.948+0000 7f08c56a9640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f08c013f650 0x7f08c00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:52303/0 (socket says 172.21.15.22:52303) 2024-03-29T11:41:24.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.948+0000 7f08c56a9640 1 -- 172.21.15.22:0/3348882634 learned_addr learned my addr 172.21.15.22:0/3348882634 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:24.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558135205 0 0) 0x7f08c00aaef0 con 0x7f08c013f650 2024-03-29T11:41:24.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ac003520 con 0x7f08c013f650 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821806344 0 0) 0x7f08c0145bd0 con 0x7f08c01457f0 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c00aaef0 con 0x7f08c01457f0 2024-03-29T11:41:24.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4262680463 0 0) 0x7f08ac003520 con 0x7f08c013f650 2024-03-29T11:41:24.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.950+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 >> v1:172.21.15.22:6790/0 conn(0x7f08c00a9d70 legacy=0x7f08c014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:24.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 >> v1:172.21.15.204:6789/0 conn(0x7f08c01457f0 legacy=0x7f08c013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 10 monclient: _finish_hunting 0 2024-03-29T11:41:24.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 1 monclient: found mon.noname-a 2024-03-29T11:41:24.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:24.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:41:24.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0146340 con 0x7f08c013f650 2024-03-29T11:41:24.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 10 monclient: _finish_auth 0 2024-03-29T11:41:24.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 10 monclient: _check_auth_tickets 2024-03-29T11:41:24.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:24.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08b8001a10 con 0x7f08c013f650 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c4ea8640 10 monclient: dump: 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:24.957 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:24.958 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:24.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c810e640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:41:24.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.951+0000 7f08c810e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:24.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.953+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08b8001bf0 con 0x7f08c013f650 2024-03-29T11:41:24.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.953+0000 7f08c4ea8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:24.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.953+0000 7f08c4ea8640 1 -- 172.21.15.22:0/3348882634 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08b8002040 con 0x7f08c013f650 2024-03-29T11:41:24.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.953+0000 7f08c4ea8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.953+0000 7f08c4ea8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:24.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.953+0000 7f08c4ea8640 10 monclient: dump: 2024-03-29T11:41:24.960 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:24.960 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:24.960 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.960 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.960 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.960 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:24.960 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.953+0000 7f08c810e640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.953+0000 7f08c810e640 4 set_mon_vals no callback set 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.954+0000 7f08c810e640 10 monclient: shutdown 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.954+0000 7f08c810e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.954+0000 7f08c810e640 1 -- 172.21.15.22:0/3348882634 >> v1:172.21.15.22:6789/0 conn(0x7f08c013f650 legacy=0x7f08c00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.954+0000 7f08c810e640 1 -- 172.21.15.22:0/3348882634 shutdown_connections 2024-03-29T11:41:24.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.954+0000 7f08c810e640 1 -- 172.21.15.22:0/3348882634 >> 172.21.15.22:0/3348882634 conn(0x7f08c0066a00 msgr2=0x7f08c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.954+0000 7f08c810e640 1 -- 172.21.15.22:0/3348882634 shutdown_connections 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.954+0000 7f08c810e640 1 -- 172.21.15.22:0/3348882634 wait complete. 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.955+0000 7f08c810e640 10 monclient: build_initial_monmap 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.955+0000 7f08c810e640 10 monclient: monmap: 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:41:24.962 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:24.963 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:24.963 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:24.963 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:41:24.963 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:41:24.963 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 1 librados: starting msgr at 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 1 librados: starting objecter 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 1 Processor -- start 2024-03-29T11:41:24.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 1 -- start start 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 1 librados: setting wanted keys 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 1 librados: calling monclient init 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 10 monclient: init 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 10 monclient(hunting): picked mon.noname-b con 0x7f08c00a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 10 monclient(hunting): picked mon.noname-a con 0x7f08c013f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:24.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 10 monclient(hunting): picked mon.noname-c con 0x7f08c01457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.956+0000 7f08c810e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c013ffb0 con 0x7f08c013f650 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.958+0000 7f08c810e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c01db250 con 0x7f08c00a9d70 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.958+0000 7f08c810e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c01415e0 con 0x7f08c01457f0 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.958+0000 7f08c810e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.958+0000 7f08c810e640 10 monclient(hunting): authenticate will time out at 1335.235397s 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.961+0000 7f08c56a9640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f08c013f650 0x7f08c01da430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:52415/0 (socket says 172.21.15.22:52415) 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.961+0000 7f08c56a9640 1 -- 172.21.15.22:0/2840780940 learned_addr learned my addr 172.21.15.22:0/2840780940 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.962+0000 7f08b67fc640 10 client.?.objecter ms_handle_connect 0x7f08c013f650 2024-03-29T11:41:24.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.962+0000 7f08b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.962+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545242544 0 0) 0x7f08c013ffb0 con 0x7f08c013f650 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.962+0000 7f08b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.962+0000 7f08b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.962+0000 7f08b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.962+0000 7f08b67fc640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.962+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a4004eb0 con 0x7f08c013f650 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 client.?.objecter ms_handle_connect 0x7f08c00a9d70 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:24.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 client.?.objecter ms_handle_connect 0x7f08c01457f0 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109284919 0 0) 0x7f08c01db250 con 0x7f08c00a9d70 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a40051d0 con 0x7f08c00a9d70 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667070012 0 0) 0x7f08c01415e0 con 0x7f08c01457f0 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:24.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:41:24.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.963+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c01db250 con 0x7f08c01457f0 2024-03-29T11:41:24.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4191877540 0 0) 0x7f08a4004eb0 con 0x7f08c013f650 2024-03-29T11:41:24.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a40057d0 con 0x7f08c013f650 2024-03-29T11:41:24.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3758252112 0 0) 0x7f08a40051d0 con 0x7f08c00a9d70 2024-03-29T11:41:24.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a4004eb0 con 0x7f08c00a9d70 2024-03-29T11:41:24.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08b8001f00 con 0x7f08c013f650 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): dump: 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.974 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08bc0017d0 con 0x7f08c00a9d70 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): dump: 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:24.975 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:24.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 881262091 0 0) 0x7f08c01db250 con 0x7f08c01457f0 2024-03-29T11:41:24.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08a40051d0 con 0x7f08c01457f0 2024-03-29T11:41:24.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08b0001490 con 0x7f08c01457f0 2024-03-29T11:41:24.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:24.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.966+0000 7f08b67fc640 10 monclient(hunting): dump: 2024-03-29T11:41:24.977 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:24.977 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:24.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 993394233 0 0) 0x7f08a40051d0 con 0x7f08c01457f0 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 >> v1:172.21.15.204:6789/0 conn(0x7f08c00a9d70 legacy=0x7f08c0142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 >> v1:172.21.15.22:6789/0 conn(0x7f08c013f650 legacy=0x7f08c01da430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 1 monclient: found mon.c 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c01418c0 con 0x7f08c01457f0 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:24.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 10 monclient: _renew_subs 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c0141ba0 con 0x7f08c01457f0 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 10 monclient: _renew_subs 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:24.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c01359a0 con 0x7f08c01457f0 2024-03-29T11:41:24.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.967+0000 7f08c810e640 1 librados: init done 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.969+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08b00019a0 con 0x7f08c01457f0 2024-03-29T11:41:24.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.969+0000 7f08b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.969+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f08b0001490 con 0x7f08c01457f0 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.969+0000 7f08b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.969+0000 7f08b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.969+0000 7f08b67fc640 10 monclient: dump: 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:24.983 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.969+0000 7f08b7fff640 4 set_mon_vals no callback set 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.970+0000 7f08abfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.970+0000 7f08abfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.970+0000 7f08abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.970+0000 7f08abfff640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a0005080 con 0x7f08c01457f0 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.973+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f08b0001c10 con 0x7f08c01457f0 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.974+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f08b00831c0 con 0x7f08c01457f0 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.974+0000 7f08b67fc640 10 client.4493.objecter ms_dispatch 0x7f08c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.974+0000 7f08b67fc640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:24.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.974+0000 7f08b67fc640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.974+0000 7f08b67fc640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.974+0000 7f08c5eaa640 10 monclient: get_auth_request con 0x7f08a406e840 auth_method 0 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.975+0000 7f08b67fc640 10 client.4493.objecter ms_handle_connect 0x7f08a406e840 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.980+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08b004eb00 con 0x7f08c01457f0 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.980+0000 7f08b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:24.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:24.980+0000 7f08b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:25.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.306+0000 7f08abfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.306+0000 7f08abfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.307+0000 7f08abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:25.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.307+0000 7f08abfff640 1 -- 172.21.15.22:0/2840780940 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08a0005330 con 0x7f08c01457f0 2024-03-29T11:41:25.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.308+0000 7f08b67fc640 1 -- 172.21.15.22:0/2840780940 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f08b004eb00 con 0x7f08c01457f0 2024-03-29T11:41:25.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.308+0000 7f08b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.308+0000 7f08b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:25.312 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:41:25.312 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:25.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.317+0000 7f08abfff640 10 librados: watch_flush enter 2024-03-29T11:41:25.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.317+0000 7f08abfff640 10 librados: watch_flush exit 2024-03-29T11:41:25.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.317+0000 7f08abfff640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:41:25.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.317+0000 7f08abfff640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:41:25.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.317+0000 7f08abfff640 1 -- 172.21.15.22:0/2840780940 >> v1:172.21.15.204:6816/3848099522 conn(0x7f08a406e840 legacy=0x7f08a4070d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.318+0000 7f08abfff640 10 monclient: shutdown 2024-03-29T11:41:25.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.318+0000 7f08abfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:25.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.318+0000 7f08abfff640 1 -- 172.21.15.22:0/2840780940 >> v1:172.21.15.22:6790/0 conn(0x7f08c01457f0 legacy=0x7f08c01dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.318+0000 7f08abfff640 1 -- 172.21.15.22:0/2840780940 shutdown_connections 2024-03-29T11:41:25.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.318+0000 7f08abfff640 1 -- 172.21.15.22:0/2840780940 >> 172.21.15.22:0/2840780940 conn(0x7f08c0066a00 msgr2=0x7f08c00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:25.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.319+0000 7f08abfff640 1 -- 172.21.15.22:0/2840780940 shutdown_connections 2024-03-29T11:41:25.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.319+0000 7f08abfff640 1 -- 172.21.15.22:0/2840780940 wait complete. 2024-03-29T11:41:25.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:25.320+0000 7f08abfff640 1 librados: shutdown 2024-03-29T11:41:25.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:32.148 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:32.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.289+0000 7f1d08296640 10 monclient: get_monmap_and_config 2024-03-29T11:41:32.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.289+0000 7f1d08296640 10 monclient: build_initial_monmap 2024-03-29T11:41:32.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.289+0000 7f1d08296640 10 monclient: monmap: 2024-03-29T11:41:32.292 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:32.292 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:32.292 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:41:32.290689+0000 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:41:32.290689+0000 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.290+0000 7f1d08296640 1 Processor -- start 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.290+0000 7f1d08296640 1 -- start start 2024-03-29T11:41:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.290+0000 7f1d08296640 10 monclient: init 2024-03-29T11:41:32.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:32.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:32.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 10 monclient(hunting): picked mon.noname-c con 0x7f1d000b2790 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:32.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 10 monclient(hunting): picked mon.noname-b con 0x7f1d000b37d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:32.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 10 monclient(hunting): picked mon.noname-a con 0x7f1d00146940 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:32.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d000b2b70 con 0x7f1d00146940 2024-03-29T11:41:32.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0006eec0 con 0x7f1d000b2790 2024-03-29T11:41:32.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0006f1a0 con 0x7f1d000b37d0 2024-03-29T11:41:32.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:32.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.291+0000 7f1d08296640 10 monclient(hunting): authenticate will time out at 1342.568896s 2024-03-29T11:41:32.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.294+0000 7f1d06833640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f1d00146940 0x7f1d00151b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:45301/0 (socket says 172.21.15.22:45301) 2024-03-29T11:41:32.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.294+0000 7f1d06833640 1 -- 172.21.15.22:0/3327950295 learned_addr learned my addr 172.21.15.22:0/3327950295 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174548075 0 0) 0x7f1d000b2b70 con 0x7f1d00146940 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cec003520 con 0x7f1d00146940 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443066099 0 0) 0x7f1d0006eec0 con 0x7f1d000b2790 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d000b2b70 con 0x7f1d000b2790 2024-03-29T11:41:32.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214952526 0 0) 0x7f1d0006f1a0 con 0x7f1d000b37d0 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d0006eec0 con 0x7f1d000b37d0 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345181639 0 0) 0x7f1d0006eec0 con 0x7f1d000b37d0 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 >> v1:172.21.15.204:6789/0 conn(0x7f1d000b2790 legacy=0x7f1d00143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 >> v1:172.21.15.22:6789/0 conn(0x7f1d00146940 legacy=0x7f1d00151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient: _finish_hunting 0 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 monclient: found mon.noname-b 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d000b2e20 con 0x7f1d000b37d0 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient: _finish_auth 0 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient: _check_auth_tickets 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d08296640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d08296640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1cf0001460 con 0x7f1d000b37d0 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient: dump: 2024-03-29T11:41:32.307 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d08296640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:32.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cf0001680 con 0x7f1d000b37d0 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d05030640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d08296640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.297+0000 7f1d08296640 4 set_mon_vals no callback set 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.298+0000 7f1d05030640 1 -- 172.21.15.22:0/3327950295 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1cf0001a10 con 0x7f1d000b37d0 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.298+0000 7f1d05030640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.298+0000 7f1d05030640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.298+0000 7f1d05030640 10 monclient: dump: 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:32.309 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.298+0000 7f1d08296640 10 monclient: shutdown 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.298+0000 7f1d08296640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.298+0000 7f1d08296640 1 -- 172.21.15.22:0/3327950295 >> v1:172.21.15.22:6790/0 conn(0x7f1d000b37d0 legacy=0x7f1d00146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.301+0000 7f1d08296640 1 -- 172.21.15.22:0/3327950295 shutdown_connections 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.301+0000 7f1d08296640 1 -- 172.21.15.22:0/3327950295 >> 172.21.15.22:0/3327950295 conn(0x7f1d0006d180 msgr2=0x7f1d00066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.302+0000 7f1d08296640 1 -- 172.21.15.22:0/3327950295 shutdown_connections 2024-03-29T11:41:32.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.302+0000 7f1d08296640 1 -- 172.21.15.22:0/3327950295 wait complete. 2024-03-29T11:41:32.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 10 monclient: build_initial_monmap 2024-03-29T11:41:32.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 10 monclient: monmap: 2024-03-29T11:41:32.319 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:32.319 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:32.319 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:41:32.319 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:41:32.319 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 1 librados: starting msgr at 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 1 librados: starting objecter 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 1 Processor -- start 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 1 -- start start 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 1 librados: setting wanted keys 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 1 librados: calling monclient init 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.303+0000 7f1d08296640 10 monclient: init 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:32.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 10 monclient(hunting): picked mon.noname-c con 0x7f1d000b2790 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 10 monclient(hunting): picked mon.noname-a con 0x7f1d000b37d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 10 monclient(hunting): picked mon.noname-b con 0x7f1d00146940 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d001d8990 con 0x7f1d000b37d0 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d001d8c50 con 0x7f1d00146940 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0013d680 con 0x7f1d000b2790 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.304+0000 7f1d08296640 10 monclient(hunting): authenticate will time out at 1342.581495s 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.307+0000 7f1d06032640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f1d000b2790 0x7f1d001d8280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42500/0 (socket says 172.21.15.22:42500) 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.307+0000 7f1d06032640 1 -- 172.21.15.22:0/3989355249 learned_addr learned my addr 172.21.15.22:0/3989355249 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.307+0000 7f1cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f1d000b2790 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.307+0000 7f1cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.308+0000 7f1cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f1d00146940 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.308+0000 7f1cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.308+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993460590 0 0) 0x7f1d0013d680 con 0x7f1d000b2790 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.308+0000 7f1cf6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.308+0000 7f1cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.308+0000 7f1cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.308+0000 7f1cf6ffd640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.308+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce4004eb0 con 0x7f1d000b2790 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.309+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961722205 0 0) 0x7f1d001d8c50 con 0x7f1d00146940 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.309+0000 7f1cf6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.309+0000 7f1cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.309+0000 7f1cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.309+0000 7f1cf6ffd640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.309+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce40051d0 con 0x7f1d00146940 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995945114 0 0) 0x7f1ce4004eb0 con 0x7f1d000b2790 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d001d8c50 con 0x7f1d000b2790 2024-03-29T11:41:32.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1cfc001870 con 0x7f1d000b2790 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:32.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3566649281 0 0) 0x7f1ce40051d0 con 0x7f1d00146940 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce4005b60 con 0x7f1d00146940 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1cf8001d00 con 0x7f1d00146940 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:32.324 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3821465136 0 0) 0x7f1d001d8c50 con 0x7f1d000b2790 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.310+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 >> v1:172.21.15.204:6789/0 conn(0x7f1d00146940 legacy=0x7f1d001db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.311+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 >> v1:172.21.15.22:6789/0 conn(0x7f1d000b37d0 legacy=0x7f1d0013be20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 1 monclient: found mon.c 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0013cbe0 con 0x7f1d000b2790 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f1d000b37d0 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.312+0000 7f1cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:41:32.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 10 monclient: _renew_subs 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0013ce90 con 0x7f1d000b2790 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 10 monclient: _renew_subs 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0013c710 con 0x7f1d000b2790 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.313+0000 7f1d08296640 1 librados: init done 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf4ff9640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0013ca10 con 0x7f1d000b2790 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cfc001d60 con 0x7f1d000b2790 2024-03-29T11:41:32.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1cfc001870 con 0x7f1d000b2790 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1cf6ffd640 10 monclient: dump: 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:32.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.315+0000 7f1d0482f640 4 set_mon_vals no callback set 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.317+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f1cfc0035f0 con 0x7f1d000b2790 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.318+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f1cfc054320 con 0x7f1d000b2790 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.318+0000 7f1cf6ffd640 10 client.4499.objecter ms_dispatch 0x7f1d00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.318+0000 7f1cf6ffd640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.318+0000 7f1cf6ffd640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.318+0000 7f1cf6ffd640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.320+0000 7f1d05831640 10 monclient: get_auth_request con 0x7f1ce406e490 auth_method 0 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.321+0000 7f1cf6ffd640 10 client.4499.objecter ms_handle_connect 0x7f1ce406e490 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.325+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1cfc0505c0 con 0x7f1d000b2790 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.325+0000 7f1cf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.325+0000 7f1cf6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:32.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.656+0000 7f1cf4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.656+0000 7f1cf4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.656+0000 7f1cf4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:32.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.656+0000 7f1cf4ff9640 1 -- 172.21.15.22:0/3989355249 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d0014a2b0 con 0x7f1d000b2790 2024-03-29T11:41:32.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.657+0000 7f1cf6ffd640 1 -- 172.21.15.22:0/3989355249 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f1cfc0505c0 con 0x7f1d000b2790 2024-03-29T11:41:32.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.657+0000 7f1cf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.657+0000 7f1cf6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:32.663 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:41:32.663 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:32.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.661+0000 7f1cf4ff9640 10 librados: watch_flush enter 2024-03-29T11:41:32.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.661+0000 7f1cf4ff9640 10 librados: watch_flush exit 2024-03-29T11:41:32.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.661+0000 7f1cf4ff9640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:41:32.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.661+0000 7f1cf4ff9640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:41:32.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.662+0000 7f1cf4ff9640 1 -- 172.21.15.22:0/3989355249 >> v1:172.21.15.204:6816/3848099522 conn(0x7f1ce406e490 legacy=0x7f1ce4070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.662+0000 7f1cf4ff9640 10 monclient: shutdown 2024-03-29T11:41:32.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.662+0000 7f1cf4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:32.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.662+0000 7f1cf4ff9640 1 -- 172.21.15.22:0/3989355249 >> v1:172.21.15.22:6790/0 conn(0x7f1d000b2790 legacy=0x7f1d001d8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.664+0000 7f1cf4ff9640 1 -- 172.21.15.22:0/3989355249 shutdown_connections 2024-03-29T11:41:32.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.664+0000 7f1cf4ff9640 1 -- 172.21.15.22:0/3989355249 >> 172.21.15.22:0/3989355249 conn(0x7f1d0006d180 msgr2=0x7f1d00066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:32.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.664+0000 7f1cf4ff9640 1 -- 172.21.15.22:0/3989355249 shutdown_connections 2024-03-29T11:41:32.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.664+0000 7f1cf4ff9640 1 -- 172.21.15.22:0/3989355249 wait complete. 2024-03-29T11:41:32.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:32.665+0000 7f1cf4ff9640 1 librados: shutdown 2024-03-29T11:41:32.684 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:39.491 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.641+0000 7fa4eb95d640 10 monclient: get_monmap_and_config 2024-03-29T11:41:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.641+0000 7fa4eb95d640 10 monclient: build_initial_monmap 2024-03-29T11:41:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.641+0000 7fa4eb95d640 10 monclient: monmap: 2024-03-29T11:41:39.644 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:39.644 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:41:39.642896+0000 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:41:39.642896+0000 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.643+0000 7fa4eb95d640 1 Processor -- start 2024-03-29T11:41:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.643+0000 7fa4eb95d640 1 -- start start 2024-03-29T11:41:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.643+0000 7fa4eb95d640 10 monclient: init 2024-03-29T11:41:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e41422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 10 monclient(hunting): picked mon.noname-c con 0x7fa4e4142690 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 10 monclient(hunting): picked mon.noname-b con 0x7fa4e40b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e4142c80 con 0x7fa4e41422b0 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e40aae40 con 0x7fa4e4142690 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e413fbd0 con 0x7fa4e40b17d0 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.644+0000 7fa4eb95d640 10 monclient(hunting): authenticate will time out at 1349.921733s 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.645+0000 7fa4e9efa640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa4e40b17d0 0x7fa4e40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53712/0 (socket says 172.21.15.22:53712) 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.645+0000 7fa4e9efa640 1 -- 172.21.15.22:0/2568343888 learned_addr learned my addr 172.21.15.22:0/2568343888 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.645+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57412101 0 0) 0x7fa4e413fbd0 con 0x7fa4e40b17d0 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.646+0000 7fa4dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.646+0000 7fa4dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.646+0000 7fa4dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:39.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.646+0000 7fa4dbfff640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:41:39.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.646+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d0003520 con 0x7fa4e40b17d0 2024-03-29T11:41:39.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.646+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2521453698 0 0) 0x7fa4d0003520 con 0x7fa4e40b17d0 2024-03-29T11:41:39.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.646+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 >> v1:172.21.15.204:6789/0 conn(0x7fa4e4142690 legacy=0x7fa4e40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.648+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 >> v1:172.21.15.22:6789/0 conn(0x7fa4e41422b0 legacy=0x7fa4e413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.648+0000 7fa4dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.648+0000 7fa4dbfff640 1 monclient: found mon.noname-b 2024-03-29T11:41:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.648+0000 7fa4dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.648+0000 7fa4dbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.649+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e4151940 con 0x7fa4e40b17d0 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.649+0000 7fa4dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.649+0000 7fa4dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.649+0000 7fa4dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.649+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa4dc001a10 con 0x7fa4e40b17d0 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.649+0000 7fa4dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.649+0000 7fa4dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.649+0000 7fa4dbfff640 10 monclient: dump: 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.653 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:39.654 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:39.654 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:39.654 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:39.654 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.650+0000 7fa4eb95d640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:41:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.650+0000 7fa4eb95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.651+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4dc001b70 con 0x7fa4e40b17d0 2024-03-29T11:41:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.651+0000 7fa4dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.651+0000 7fa4dbfff640 1 -- 172.21.15.22:0/2568343888 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa4dc001fc0 con 0x7fa4e40b17d0 2024-03-29T11:41:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.651+0000 7fa4eb95d640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.651+0000 7fa4eb95d640 4 set_mon_vals no callback set 2024-03-29T11:41:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.652+0000 7fa4eb95d640 10 monclient: shutdown 2024-03-29T11:41:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.652+0000 7fa4eb95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.652+0000 7fa4eb95d640 1 -- 172.21.15.22:0/2568343888 >> v1:172.21.15.22:6790/0 conn(0x7fa4e40b17d0 legacy=0x7fa4e40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.652+0000 7fa4dbfff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:41:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.652+0000 7fa4eb95d640 1 -- 172.21.15.22:0/2568343888 shutdown_connections 2024-03-29T11:41:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.652+0000 7fa4eb95d640 1 -- 172.21.15.22:0/2568343888 >> 172.21.15.22:0/2568343888 conn(0x7fa4e4066a00 msgr2=0x7fa4e4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.653+0000 7fa4eb95d640 1 -- 172.21.15.22:0/2568343888 shutdown_connections 2024-03-29T11:41:39.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.653+0000 7fa4eb95d640 1 -- 172.21.15.22:0/2568343888 wait complete. 2024-03-29T11:41:39.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.654+0000 7fa4eb95d640 10 monclient: build_initial_monmap 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.654+0000 7fa4eb95d640 10 monclient: monmap: 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.654+0000 7fa4eb95d640 1 librados: starting msgr at 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.654+0000 7fa4eb95d640 1 librados: starting objecter 2024-03-29T11:41:39.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.654+0000 7fa4eb95d640 1 Processor -- start 2024-03-29T11:41:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.655+0000 7fa4eb95d640 1 -- start start 2024-03-29T11:41:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.655+0000 7fa4eb95d640 1 librados: setting wanted keys 2024-03-29T11:41:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.655+0000 7fa4eb95d640 1 librados: calling monclient init 2024-03-29T11:41:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.655+0000 7fa4eb95d640 10 monclient: init 2024-03-29T11:41:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 10 monclient(hunting): picked mon.noname-b con 0x7fa4e40b17d0 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e41422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 10 monclient(hunting): picked mon.noname-c con 0x7fa4e4142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e406ea40 con 0x7fa4e41422b0 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e406ecb0 con 0x7fa4e40b17d0 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e412f680 con 0x7fa4e4142690 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4e9efa640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa4e4142690 0x7fa4e412ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55168/0 (socket says 172.21.15.22:55168) 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4e9efa640 1 -- 172.21.15.22:0/4150270851 learned_addr learned my addr 172.21.15.22:0/4150270851 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.656+0000 7fa4eb95d640 10 monclient(hunting): authenticate will time out at 1349.933316s 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 10 client.?.objecter ms_handle_connect 0x7fa4e4142690 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002911439 0 0) 0x7fa4e412f680 con 0x7fa4e4142690 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c8004eb0 con 0x7fa4e4142690 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 10 client.?.objecter ms_handle_connect 0x7fa4e41422b0 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.659+0000 7fa4d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502472389 0 0) 0x7fa4e406ea40 con 0x7fa4e41422b0 2024-03-29T11:41:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e412f680 con 0x7fa4e41422b0 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1421692113 0 0) 0x7fa4c8004eb0 con 0x7fa4e4142690 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4e406ea40 con 0x7fa4e4142690 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2761959625 0 0) 0x7fa4e412f680 con 0x7fa4e41422b0 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4c8004eb0 con 0x7fa4e41422b0 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa4dc001f40 con 0x7fa4e4142690 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.660+0000 7fa4d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:39.668 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa4e0001710 con 0x7fa4e41422b0 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:39.669 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 141867785 0 0) 0x7fa4e406ea40 con 0x7fa4e4142690 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 >> v1:172.21.15.204:6789/0 conn(0x7fa4e40b17d0 legacy=0x7fa4e4150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 >> v1:172.21.15.22:6789/0 conn(0x7fa4e41422b0 legacy=0x7fa4e412e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:39.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 1 monclient: found mon.c 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e406ef90 con 0x7fa4e4142690 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.661+0000 7fa4d9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4d9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 10 monclient: _renew_subs 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e41505d0 con 0x7fa4e4142690 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 10 monclient: _renew_subs 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:39.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e41dec50 con 0x7fa4e4142690 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4eb95d640 1 librados: init done 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4dc004f70 con 0x7fa4e4142690 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa4dc005280 con 0x7fa4e4142690 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.662+0000 7fa4d9ffb640 10 monclient: dump: 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:39.672 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:39.673 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:39.673 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:39.673 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:39.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.663+0000 7fa4db7fe640 4 set_mon_vals no callback set 2024-03-29T11:41:39.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fa4dc01b250 con 0x7fa4e4142690 2024-03-29T11:41:39.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4eb95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4eb95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4eb95d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:39.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c4005080 con 0x7fa4e4142690 2024-03-29T11:41:39.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fa4dc084f10 con 0x7fa4e4142690 2024-03-29T11:41:39.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4d9ffb640 10 client.4505.objecter ms_dispatch 0x7fa4e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:39.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4d9ffb640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:39.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4d9ffb640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:39.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.665+0000 7fa4d9ffb640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:41:39.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.667+0000 7fa4e96f9640 10 monclient: get_auth_request con 0x7fa4c806df20 auth_method 0 2024-03-29T11:41:39.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.668+0000 7fa4d9ffb640 10 client.4505.objecter ms_handle_connect 0x7fa4c806df20 2024-03-29T11:41:39.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.674+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4dc0507f0 con 0x7fa4e4142690 2024-03-29T11:41:39.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.674+0000 7fa4d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:39.674+0000 7fa4d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:40.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.005+0000 7fa4eb95d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.005+0000 7fa4eb95d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.005+0000 7fa4eb95d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:40.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.005+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4c4005330 con 0x7fa4e4142690 2024-03-29T11:41:40.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.006+0000 7fa4d9ffb640 1 -- 172.21.15.22:0/4150270851 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fa4dc052720 con 0x7fa4e4142690 2024-03-29T11:41:40.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.006+0000 7fa4d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.006+0000 7fa4d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:40.010 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:41:40.010 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:40.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.010+0000 7fa4eb95d640 10 librados: watch_flush enter 2024-03-29T11:41:40.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.010+0000 7fa4eb95d640 10 librados: watch_flush exit 2024-03-29T11:41:40.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.010+0000 7fa4eb95d640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:41:40.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.010+0000 7fa4eb95d640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:41:40.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 >> v1:172.21.15.204:6816/3848099522 conn(0x7fa4c806df20 legacy=0x7fa4c80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 10 monclient: shutdown 2024-03-29T11:41:40.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:40.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 >> v1:172.21.15.22:6790/0 conn(0x7fa4e4142690 legacy=0x7fa4e412ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 shutdown_connections 2024-03-29T11:41:40.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 >> 172.21.15.22:0/4150270851 conn(0x7fa4e4066a00 msgr2=0x7fa4e40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:40.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 shutdown_connections 2024-03-29T11:41:40.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 1 -- 172.21.15.22:0/4150270851 wait complete. 2024-03-29T11:41:40.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:40.011+0000 7fa4eb95d640 1 librados: shutdown 2024-03-29T11:41:40.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:46.837 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.989+0000 7feb31915640 10 monclient: get_monmap_and_config 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.989+0000 7feb31915640 10 monclient: build_initial_monmap 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.990+0000 7feb31915640 10 monclient: monmap: 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:41:46.991261+0000 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:41:46.991261+0000 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:46.993 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:46.994 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:41:46.994 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:41:46.994 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:46.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.991+0000 7feb31915640 1 Processor -- start 2024-03-29T11:41:46.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.991+0000 7feb31915640 1 -- start start 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.991+0000 7feb31915640 10 monclient: init 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 10 monclient(hunting): picked mon.noname-a con 0x7feb24006a70 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 10 monclient(hunting): picked mon.noname-c con 0x7feb2402b100 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 10 monclient(hunting): picked mon.noname-b con 0x7feb24029c30 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2402a270 con 0x7feb24006a70 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2402c440 con 0x7feb2402b100 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb240c9440 con 0x7feb24029c30 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:46.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.992+0000 7feb31915640 10 monclient(hunting): authenticate will time out at 1357.269320s 2024-03-29T11:41:46.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.995+0000 7feb31114640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7feb24029c30 0x7feb240d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39035/0 (socket says 172.21.15.22:39035) 2024-03-29T11:41:46.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.995+0000 7feb31114640 1 -- 172.21.15.22:0/3714658883 learned_addr learned my addr 172.21.15.22:0/3714658883 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:47.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.997+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202319449 0 0) 0x7feb240c9440 con 0x7feb24029c30 2024-03-29T11:41:47.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.997+0000 7feb2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.997+0000 7feb2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.997+0000 7feb2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.997+0000 7feb2b7fe640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:41:47.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.997+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb1c003520 con 0x7feb24029c30 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.997+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2728473581 0 0) 0x7feb1c003520 con 0x7feb24029c30 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.997+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 >> v1:172.21.15.204:6789/0 conn(0x7feb2402b100 legacy=0x7feb240cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 >> v1:172.21.15.22:6789/0 conn(0x7feb24006a70 legacy=0x7feb240c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240c95d0 con 0x7feb24029c30 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:47.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7feb2c04f730 con 0x7feb24029c30 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb2b7fe640 10 monclient: dump: 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:47.003 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:47.004 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:47.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb31915640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:41:47.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:46.998+0000 7feb31915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:47.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.002+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb2c0008d0 con 0x7feb24029c30 2024-03-29T11:41:47.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.002+0000 7feb2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:47.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.002+0000 7feb2b7fe640 1 -- 172.21.15.22:0/3714658883 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7feb2c0758b0 con 0x7feb24029c30 2024-03-29T11:41:47.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.002+0000 7feb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.002+0000 7feb2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.002+0000 7feb2b7fe640 10 monclient: dump: 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.002+0000 7feb31915640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:47.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.002+0000 7feb31915640 4 set_mon_vals no callback set 2024-03-29T11:41:47.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.003+0000 7feb31915640 10 monclient: shutdown 2024-03-29T11:41:47.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.003+0000 7feb31915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:47.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.003+0000 7feb31915640 1 -- 172.21.15.22:0/3714658883 >> v1:172.21.15.22:6790/0 conn(0x7feb24029c30 legacy=0x7feb240d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.004+0000 7feb31915640 1 -- 172.21.15.22:0/3714658883 shutdown_connections 2024-03-29T11:41:47.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.004+0000 7feb31915640 1 -- 172.21.15.22:0/3714658883 >> 172.21.15.22:0/3714658883 conn(0x7feb240058f0 msgr2=0x7feb24005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:47.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.007+0000 7feb31915640 1 -- 172.21.15.22:0/3714658883 shutdown_connections 2024-03-29T11:41:47.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.007+0000 7feb31915640 1 -- 172.21.15.22:0/3714658883 wait complete. 2024-03-29T11:41:47.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 10 monclient: build_initial_monmap 2024-03-29T11:41:47.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 10 monclient: monmap: 2024-03-29T11:41:47.014 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:47.014 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:47.014 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:41:47.014 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:41:47.014 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 1 librados: starting msgr at 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 1 librados: starting objecter 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 1 Processor -- start 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 1 -- start start 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 1 librados: setting wanted keys 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 1 librados: calling monclient init 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.008+0000 7feb31915640 10 monclient: init 2024-03-29T11:41:47.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.009+0000 7feb31915640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.009+0000 7feb31915640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.010+0000 7feb31915640 10 monclient(hunting): picked mon.noname-a con 0x7feb24006a70 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.010+0000 7feb31915640 10 monclient(hunting): picked mon.noname-b con 0x7feb24029c30 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.010+0000 7feb31915640 10 monclient(hunting): picked mon.noname-c con 0x7feb24032310 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.010+0000 7feb31915640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb24160aa0 con 0x7feb24006a70 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.010+0000 7feb31915640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2402ee20 con 0x7feb24029c30 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.010+0000 7feb31915640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2402f090 con 0x7feb24032310 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.010+0000 7feb31915640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:47.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.010+0000 7feb31915640 10 monclient(hunting): authenticate will time out at 1357.287122s 2024-03-29T11:41:47.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.012+0000 7feb31114640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7feb24032310 0x7feb2402d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39163/0 (socket says 172.21.15.22:39163) 2024-03-29T11:41:47.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.012+0000 7feb31114640 1 -- 172.21.15.22:0/191157537 learned_addr learned my addr 172.21.15.22:0/191157537 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:47.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.012+0000 7feb297fa640 10 client.?.objecter ms_handle_connect 0x7feb24032310 2024-03-29T11:41:47.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.012+0000 7feb297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581948968 0 0) 0x7feb2402f090 con 0x7feb24032310 2024-03-29T11:41:47.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:41:47.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb10004eb0 con 0x7feb24032310 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1426790412 0 0) 0x7feb10004eb0 con 0x7feb24032310 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb2402f090 con 0x7feb24032310 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7feb2c04f400 con 0x7feb24032310 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 10 monclient(hunting): dump: 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:47.019 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:47.020 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:47.020 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:47.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:47.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2638645335 0 0) 0x7feb2402f090 con 0x7feb24032310 2024-03-29T11:41:47.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.013+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 >> v1:172.21.15.204:6789/0 conn(0x7feb24029c30 legacy=0x7feb24160f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:47.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 >> v1:172.21.15.22:6789/0 conn(0x7feb24006a70 legacy=0x7feb24031c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 1 monclient: found mon.c 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240ccbf0 con 0x7feb24032310 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 10 client.?.objecter ms_handle_connect 0x7feb24029c30 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 10 monclient: _renew_subs 2024-03-29T11:41:47.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb24161640 con 0x7feb24032310 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 10 monclient: _renew_subs 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2402dce0 con 0x7feb24032310 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.020+0000 7feb31915640 1 librados: init done 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.022+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb2c04fdd0 con 0x7feb24032310 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.022+0000 7feb297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.022+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7feb2c067bf0 con 0x7feb24032310 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.022+0000 7feb297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.022+0000 7feb297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.022+0000 7feb297fa640 10 monclient: dump: 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:47.026 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:47.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.022+0000 7feb2affd640 4 set_mon_vals no callback set 2024-03-29T11:41:47.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.023+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7feb2c067e10 con 0x7feb24032310 2024-03-29T11:41:47.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.024+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7feb2c0ff680 con 0x7feb24032310 2024-03-29T11:41:47.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.024+0000 7feb297fa640 10 client.4511.objecter ms_dispatch 0x7feb2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:47.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.024+0000 7feb297fa640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:47.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.024+0000 7feb297fa640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:47.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.024+0000 7feb297fa640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T11:41:47.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.024+0000 7feb30913640 10 monclient: get_auth_request con 0x7feb1006da80 auth_method 0 2024-03-29T11:41:47.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.025+0000 7feb297fa640 10 client.4511.objecter ms_handle_connect 0x7feb1006da80 2024-03-29T11:41:47.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.025+0000 7feb31915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.025+0000 7feb31915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.025+0000 7feb31915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:47.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.025+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c005080 con 0x7feb24032310 2024-03-29T11:41:47.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.034+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb2c0caf60 con 0x7feb24032310 2024-03-29T11:41:47.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.034+0000 7feb297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.034+0000 7feb297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:47.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.361+0000 7feb31915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.361+0000 7feb31915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.361+0000 7feb31915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:41:47.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.361+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb0c005330 con 0x7feb24032310 2024-03-29T11:41:47.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.362+0000 7feb297fa640 1 -- 172.21.15.22:0/191157537 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7feb2c0cce90 con 0x7feb24032310 2024-03-29T11:41:47.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.362+0000 7feb297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.362+0000 7feb297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:47.365 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:41:47.365 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:47.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.365+0000 7feb31915640 10 librados: watch_flush enter 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.365+0000 7feb31915640 10 librados: watch_flush exit 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.365+0000 7feb31915640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.365+0000 7feb31915640 10 client.4511.objecter successfully canceled tick 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.365+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 >> v1:172.21.15.204:6816/3848099522 conn(0x7feb1006da80 legacy=0x7feb1006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.365+0000 7feb31915640 10 monclient: shutdown 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.365+0000 7feb31915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.365+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 >> v1:172.21.15.22:6790/0 conn(0x7feb24032310 legacy=0x7feb2402d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.366+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 shutdown_connections 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.366+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 >> 172.21.15.22:0/191157537 conn(0x7feb240058f0 msgr2=0x7feb240c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.366+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 shutdown_connections 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.366+0000 7feb31915640 1 -- 172.21.15.22:0/191157537 wait complete. 2024-03-29T11:41:47.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:47.366+0000 7feb31915640 1 librados: shutdown 2024-03-29T11:41:47.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:54.193 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:54.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.341+0000 7ffaf3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:41:54.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.341+0000 7ffaf3fff640 10 monclient: build_initial_monmap 2024-03-29T11:41:54.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.341+0000 7ffaf3fff640 10 monclient: monmap: 2024-03-29T11:41:54.345 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:54.345 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:54.346 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:41:54.342397+0000 2024-03-29T11:41:54.346 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:41:54.342397+0000 2024-03-29T11:41:54.346 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:54.346 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:54.346 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:54.346 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:41:54.346 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:41:54.346 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:54.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.344+0000 7ffaf3fff640 1 Processor -- start 2024-03-29T11:41:54.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.344+0000 7ffaf3fff640 1 -- start start 2024-03-29T11:41:54.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.344+0000 7ffaf3fff640 10 monclient: init 2024-03-29T11:41:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 10 monclient(hunting): picked mon.noname-c con 0x7ffaf40ae220 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 10 monclient(hunting): picked mon.noname-b con 0x7ffaf4072f10 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf40acea0 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40aa9f0 con 0x7ffaf40acea0 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40ac4c0 con 0x7ffaf40ae220 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40ac7a0 con 0x7ffaf4072f10 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.345+0000 7ffaf3fff640 10 monclient(hunting): authenticate will time out at 1364.622246s 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.348+0000 7ffaf37fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ffaf40acea0 0x7ffaf40ad2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44661/0 (socket says 172.21.15.22:44661) 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.348+0000 7ffaf37fe640 1 -- 172.21.15.22:0/1862858733 learned_addr learned my addr 172.21.15.22:0/1862858733 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.352+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595219837 0 0) 0x7ffaf40ac4c0 con 0x7ffaf40ae220 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.352+0000 7ffaf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.352+0000 7ffaf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.352+0000 7ffaf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.352+0000 7ffaf1ffb640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.352+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffadc003520 con 0x7ffaf40ae220 2024-03-29T11:41:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436776346 0 0) 0x7ffaf40aa9f0 con 0x7ffaf40acea0 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf40ac4c0 con 0x7ffaf40acea0 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885407030 0 0) 0x7ffadc003520 con 0x7ffaf40ae220 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 >> v1:172.21.15.22:6790/0 conn(0x7ffaf4072f10 legacy=0x7ffaf4141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 >> v1:172.21.15.22:6789/0 conn(0x7ffaf40acea0 legacy=0x7ffaf40ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 1 monclient: found mon.noname-c 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40ab5a0 con 0x7ffaf40ae220 2024-03-29T11:41:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ffaec001a10 con 0x7ffaf40ae220 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf1ffb640 10 monclient: dump: 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.360 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf3fff640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.353+0000 7ffaf3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.354+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaec001b70 con 0x7ffaf40ae220 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.354+0000 7ffaf1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.354+0000 7ffaf1ffb640 1 -- 172.21.15.22:0/1862858733 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ffaec001f80 con 0x7ffaf40ae220 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.354+0000 7ffaf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.354+0000 7ffaf1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:54.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.354+0000 7ffaf1ffb640 10 monclient: dump: 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:54.362 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.356+0000 7ffaf3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.356+0000 7ffaf3fff640 4 set_mon_vals no callback set 2024-03-29T11:41:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.357+0000 7ffaf3fff640 10 monclient: shutdown 2024-03-29T11:41:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.357+0000 7ffaf3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:54.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.357+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1862858733 >> v1:172.21.15.204:6789/0 conn(0x7ffaf40ae220 legacy=0x7ffaf4141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.361+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1862858733 shutdown_connections 2024-03-29T11:41:54.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.361+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1862858733 >> 172.21.15.22:0/1862858733 conn(0x7ffaf40b3a30 msgr2=0x7ffaf40b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:54.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.361+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1862858733 shutdown_connections 2024-03-29T11:41:54.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.361+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1862858733 wait complete. 2024-03-29T11:41:54.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.362+0000 7ffaf3fff640 10 monclient: build_initial_monmap 2024-03-29T11:41:54.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.362+0000 7ffaf3fff640 10 monclient: monmap: 2024-03-29T11:41:54.365 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:41:54.365 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:54.365 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:41:54.365 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:41:54.366 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:54.366 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:54.366 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:41:54.366 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:41:54.366 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:41:54.366 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:54.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.363+0000 7ffaf3fff640 1 librados: starting msgr at 2024-03-29T11:41:54.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.363+0000 7ffaf3fff640 1 librados: starting objecter 2024-03-29T11:41:54.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.363+0000 7ffaf3fff640 1 Processor -- start 2024-03-29T11:41:54.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.364+0000 7ffaf3fff640 1 -- start start 2024-03-29T11:41:54.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.364+0000 7ffaf3fff640 1 librados: setting wanted keys 2024-03-29T11:41:54.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.364+0000 7ffaf3fff640 1 librados: calling monclient init 2024-03-29T11:41:54.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.364+0000 7ffaf3fff640 10 monclient: init 2024-03-29T11:41:54.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.365+0000 7ffaf3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:54.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.365+0000 7ffaf3fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:54.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.369+0000 7ffaf3fff640 10 monclient(hunting): picked mon.noname-a con 0x7ffaf4072f10 addr v1:172.21.15.22:6789/0 2024-03-29T11:41:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.369+0000 7ffaf3fff640 10 monclient(hunting): picked mon.noname-c con 0x7ffaf40acea0 addr v1:172.21.15.22:6790/0 2024-03-29T11:41:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.371+0000 7ffaf2ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ffaf4072f10 0x7ffaf4071220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44997/0 (socket says 172.21.15.22:44997) 2024-03-29T11:41:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.371+0000 7ffaf2ffd640 1 -- 172.21.15.22:0/1280422425 learned_addr learned my addr 172.21.15.22:0/1280422425 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:41:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.371+0000 7ffadb7fe640 10 client.?.objecter ms_handle_connect 0x7ffaf4072f10 2024-03-29T11:41:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.371+0000 7ffadb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.371+0000 7ffaf3fff640 10 monclient(hunting): picked mon.noname-b con 0x7ffaf40ae220 addr v1:172.21.15.204:6789/0 2024-03-29T11:41:54.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.375+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf4141f90 con 0x7ffaf4072f10 2024-03-29T11:41:54.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.375+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf4141bd0 con 0x7ffaf40ae220 2024-03-29T11:41:54.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.375+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf406f190 con 0x7ffaf40acea0 2024-03-29T11:41:54.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.375+0000 7ffaf3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:54.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.375+0000 7ffaf3fff640 10 monclient(hunting): authenticate will time out at 1364.652947s 2024-03-29T11:41:54.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 client.?.objecter ms_handle_connect 0x7ffaf40ae220 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 client.?.objecter ms_handle_connect 0x7ffaf40acea0 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092909292 0 0) 0x7ffaf4141f90 con 0x7ffaf4072f10 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae0004eb0 con 0x7ffaf4072f10 2024-03-29T11:41:54.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781244391 0 0) 0x7ffaf4141bd0 con 0x7ffaf40ae220 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.376+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf4141f90 con 0x7ffaf40ae220 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842325717 0 0) 0x7ffaf406f190 con 0x7ffaf40acea0 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:41:54.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae0005360 con 0x7ffaf40acea0 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72371306 0 0) 0x7ffae0004eb0 con 0x7ffaf4072f10 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaf406f190 con 0x7ffaf4072f10 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ffaec001fc0 con 0x7ffaf4072f10 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): dump: 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.382 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486628310 0 0) 0x7ffaf4141f90 con 0x7ffaf40ae220 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffae0004eb0 con 0x7ffaf40ae220 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ffae80018f0 con 0x7ffaf40ae220 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 10 monclient(hunting): dump: 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:54.383 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.384 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.384 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.384 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:54.384 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:54.384 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:54.384 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:54.384 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:54.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 707960740 0 0) 0x7ffae0004eb0 con 0x7ffaf40ae220 2024-03-29T11:41:54.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.377+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 >> v1:172.21.15.22:6790/0 conn(0x7ffaf40acea0 legacy=0x7ffaf4142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 >> v1:172.21.15.22:6789/0 conn(0x7ffaf4072f10 legacy=0x7ffaf4071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 1 monclient: found mon.b 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf406f320 con 0x7ffaf40ae220 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffadb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 10 monclient: _renew_subs 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:54.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf40704a0 con 0x7ffaf40ae220 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 10 monclient: _renew_subs 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf4142c30 con 0x7ffaf40ae220 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.378+0000 7ffaf3fff640 1 librados: init done 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.379+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae8001d00 con 0x7ffaf40ae220 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.379+0000 7ffadb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.380+0000 7ffaf17fa640 4 set_mon_vals no callback set 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.380+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ffae80018f0 con 0x7ffaf40ae220 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.380+0000 7ffadb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.380+0000 7ffad97fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.380+0000 7ffadb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.380+0000 7ffadb7fe640 10 monclient: dump: 2024-03-29T11:41:54.387 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.382+0000 7ffad97fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.382+0000 7ffad97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:54.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.382+0000 7ffad97fa640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf4141d60 con 0x7ffaf40ae220 2024-03-29T11:41:54.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.383+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7ffae8004860 con 0x7ffaf40ae220 2024-03-29T11:41:54.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.383+0000 7ffaf2ffd640 10 monclient: get_auth_request con 0x7ffae006e4d0 auth_method 0 2024-03-29T11:41:54.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.383+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7ffae8001710 con 0x7ffaf40ae220 2024-03-29T11:41:54.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.384+0000 7ffadb7fe640 10 client.4459.objecter ms_dispatch 0x7ffaf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:54.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.384+0000 7ffadb7fe640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:54.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.384+0000 7ffadb7fe640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:54.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.384+0000 7ffadb7fe640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:41:54.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.384+0000 7ffadb7fe640 10 client.4459.objecter ms_handle_connect 0x7ffae006e4d0 2024-03-29T11:41:54.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.392+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffae8050450 con 0x7ffaf40ae220 2024-03-29T11:41:54.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.392+0000 7ffadb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.392+0000 7ffadb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:54.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.732+0000 7ffad97fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.732+0000 7ffad97fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.732+0000 7ffad97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:41:54.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.732+0000 7ffad97fa640 1 -- 172.21.15.22:0/1280422425 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaf4141d60 con 0x7ffaf40ae220 2024-03-29T11:41:54.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.735+0000 7ffadb7fe640 1 -- 172.21.15.22:0/1280422425 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7ffae8050450 con 0x7ffaf40ae220 2024-03-29T11:41:54.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.735+0000 7ffadb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.735+0000 7ffadb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:54.738 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:41:54.738 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:54.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.741+0000 7ffad97fa640 10 librados: watch_flush enter 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.742+0000 7ffad97fa640 10 librados: watch_flush exit 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.742+0000 7ffad97fa640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.742+0000 7ffad97fa640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.743+0000 7ffad97fa640 1 -- 172.21.15.22:0/1280422425 >> v1:172.21.15.204:6816/3848099522 conn(0x7ffae006e4d0 legacy=0x7ffae0070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.743+0000 7ffad97fa640 10 monclient: shutdown 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.743+0000 7ffad97fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.743+0000 7ffad97fa640 1 -- 172.21.15.22:0/1280422425 >> v1:172.21.15.204:6789/0 conn(0x7ffaf40ae220 legacy=0x7ffaf41e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.744+0000 7ffad97fa640 1 -- 172.21.15.22:0/1280422425 shutdown_connections 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.744+0000 7ffad97fa640 1 -- 172.21.15.22:0/1280422425 >> 172.21.15.22:0/1280422425 conn(0x7ffaf40b3a30 msgr2=0x7ffaf413de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.745+0000 7ffad97fa640 1 -- 172.21.15.22:0/1280422425 shutdown_connections 2024-03-29T11:41:54.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.745+0000 7ffad97fa640 1 -- 172.21.15.22:0/1280422425 wait complete. 2024-03-29T11:41:54.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:41:54.745+0000 7ffad97fa640 1 librados: shutdown 2024-03-29T11:41:54.764 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:01.571 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:01.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.732+0000 7f2ccbc0e640 10 monclient: get_monmap_and_config 2024-03-29T11:42:01.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.732+0000 7f2ccbc0e640 10 monclient: build_initial_monmap 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.732+0000 7f2ccbc0e640 10 monclient: monmap: 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:42:01.733539+0000 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:42:01.733539+0000 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.733+0000 7f2ccbc0e640 1 Processor -- start 2024-03-29T11:42:01.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.735+0000 7f2ccbc0e640 1 -- start start 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.735+0000 7f2ccbc0e640 10 monclient: init 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc40a9750 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc4149ac0 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc414fe50 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc40aa4a0 con 0x7f2cc414fe50 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc414a680 con 0x7f2cc40a9750 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc40af290 con 0x7f2cc4149ac0 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.736+0000 7f2ccbc0e640 10 monclient(hunting): authenticate will time out at 1372.013507s 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.738+0000 7f2cc91a9640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2cc4149ac0 0x7f2cc4149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:54492/0 (socket says 172.21.15.22:54492) 2024-03-29T11:42:01.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.738+0000 7f2cc91a9640 1 -- 172.21.15.22:0/1718566663 learned_addr learned my addr 172.21.15.22:0/1718566663 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456077566 0 0) 0x7f2cc40af290 con 0x7f2cc4149ac0 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb0003520 con 0x7f2cc4149ac0 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 297616715 0 0) 0x7f2cb0003520 con 0x7f2cc4149ac0 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 >> v1:172.21.15.204:6789/0 conn(0x7f2cc40a9750 legacy=0x7f2cc41436f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.740+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 >> v1:172.21.15.22:6789/0 conn(0x7f2cc414fe50 legacy=0x7f2cc4152240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:01.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 10 monclient: _finish_hunting 0 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 1 monclient: found mon.noname-b 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc40afb40 con 0x7f2cc4149ac0 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 10 monclient: _finish_auth 0 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 10 monclient: _check_auth_tickets 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2cb8001a10 con 0x7f2cc4149ac0 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2cc89a8640 10 monclient: dump: 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:01.746 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2ccbc0e640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.741+0000 7f2ccbc0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.742+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb8001be0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.742+0000 7f2cc89a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.742+0000 7f2cc89a8640 1 -- 172.21.15.22:0/1718566663 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2cb8002030 con 0x7f2cc4149ac0 2024-03-29T11:42:01.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.742+0000 7f2cc89a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.742+0000 7f2cc89a8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.742+0000 7f2cc89a8640 10 monclient: dump: 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.742+0000 7f2ccbc0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:01.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.742+0000 7f2ccbc0e640 4 set_mon_vals no callback set 2024-03-29T11:42:01.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.743+0000 7f2ccbc0e640 10 monclient: shutdown 2024-03-29T11:42:01.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.743+0000 7f2ccbc0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:01.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.743+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/1718566663 >> v1:172.21.15.22:6790/0 conn(0x7f2cc4149ac0 legacy=0x7f2cc4149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.743+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/1718566663 shutdown_connections 2024-03-29T11:42:01.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.743+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/1718566663 >> 172.21.15.22:0/1718566663 conn(0x7f2cc40b35e0 msgr2=0x7f2cc40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:01.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.743+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/1718566663 shutdown_connections 2024-03-29T11:42:01.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.744+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/1718566663 wait complete. 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.744+0000 7f2ccbc0e640 10 monclient: build_initial_monmap 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.744+0000 7f2ccbc0e640 10 monclient: monmap: 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.745+0000 7f2ccbc0e640 1 librados: starting msgr at 2024-03-29T11:42:01.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.745+0000 7f2ccbc0e640 1 librados: starting objecter 2024-03-29T11:42:01.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.745+0000 7f2ccbc0e640 1 Processor -- start 2024-03-29T11:42:01.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.745+0000 7f2ccbc0e640 1 -- start start 2024-03-29T11:42:01.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.745+0000 7f2ccbc0e640 1 librados: setting wanted keys 2024-03-29T11:42:01.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.745+0000 7f2ccbc0e640 1 librados: calling monclient init 2024-03-29T11:42:01.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.745+0000 7f2ccbc0e640 10 monclient: init 2024-03-29T11:42:01.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.746+0000 7f2ccbc0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:01.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.746+0000 7f2ccbc0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:01.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2ccbc0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc40a9750 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:01.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2ccbc0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc4149ac0 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:01.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2ccbc0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc414fe50 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:01.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2ccbc0e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc41dc6a0 con 0x7f2cc40a9750 2024-03-29T11:42:01.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2ccbc0e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc41dba70 con 0x7f2cc414fe50 2024-03-29T11:42:01.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2ccbc0e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc413cbd0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2ccbc0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:01.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2ccbc0e640 10 monclient(hunting): authenticate will time out at 1372.027334s 2024-03-29T11:42:01.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2cc91a9640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2cc4149ac0 0x7f2cc413a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55602/0 (socket says 172.21.15.22:55602) 2024-03-29T11:42:01.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.750+0000 7f2cc91a9640 1 -- 172.21.15.22:0/3432854503 learned_addr learned my addr 172.21.15.22:0/3432854503 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:01.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.751+0000 7f2cb67fc640 10 client.?.objecter ms_handle_connect 0x7f2cc40a9750 2024-03-29T11:42:01.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.751+0000 7f2cb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:01.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 client.?.objecter ms_handle_connect 0x7f2cc4149ac0 2024-03-29T11:42:01.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458500454 0 0) 0x7f2cc413cbd0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca8004eb0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009721959 0 0) 0x7f2cc41dc6a0 con 0x7f2cc40a9750 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.752+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc413cbd0 con 0x7f2cc40a9750 2024-03-29T11:42:01.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 client.?.objecter ms_handle_connect 0x7f2cc414fe50 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2473350559 0 0) 0x7f2ca8004eb0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cc41dc6a0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72948128 0 0) 0x7f2cc413cbd0 con 0x7f2cc40a9750 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ca8004eb0 con 0x7f2cc40a9750 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2cb8001ef0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 monclient(hunting): dump: 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.761 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2cc00017b0 con 0x7f2cc40a9750 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 10 monclient(hunting): dump: 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:01.762 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1606996460 0 0) 0x7f2cc41dc6a0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 >> v1:172.21.15.204:6789/0 conn(0x7f2cc414fe50 legacy=0x7f2cc41e70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.753+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 >> v1:172.21.15.22:6789/0 conn(0x7f2cc40a9750 legacy=0x7f2cc41dbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 1 monclient: found mon.c 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:01.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc413cd60 con 0x7f2cc4149ac0 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 10 monclient: _renew_subs 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc413c600 con 0x7f2cc4149ac0 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 10 monclient: _renew_subs 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc40ab6c0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2ccbc0e640 1 librados: init done 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb8004f70 con 0x7f2cc4149ac0 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb77fe640 4 set_mon_vals no callback set 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2cb8002050 con 0x7f2cc4149ac0 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.754+0000 7f2cb67fc640 10 monclient: dump: 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:01.765 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:01.766 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:01.766 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:01.766 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.756+0000 7f2ccbc0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.756+0000 7f2ccbc0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.756+0000 7f2ccbc0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.756+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cac005080 con 0x7f2cc4149ac0 2024-03-29T11:42:01.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.756+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f2cb801b0f0 con 0x7f2cc4149ac0 2024-03-29T11:42:01.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.757+0000 7f2cc99aa640 10 monclient: get_auth_request con 0x7f2ca806e000 auth_method 0 2024-03-29T11:42:01.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.757+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f2cb8084f60 con 0x7f2cc4149ac0 2024-03-29T11:42:01.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.757+0000 7f2cb67fc640 10 client.4523.objecter ms_dispatch 0x7f2cc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:01.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.757+0000 7f2cb67fc640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:01.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.757+0000 7f2cb67fc640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:01.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.757+0000 7f2cb67fc640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T11:42:01.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.757+0000 7f2cb67fc640 10 client.4523.objecter ms_handle_connect 0x7f2ca806e000 2024-03-29T11:42:01.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.767+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2cb8050840 con 0x7f2cc4149ac0 2024-03-29T11:42:01.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.767+0000 7f2cb67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:01.767+0000 7f2cb67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:02.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.126+0000 7f2ccbc0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.126+0000 7f2ccbc0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.126+0000 7f2ccbc0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:02.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.126+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cac005330 con 0x7f2cc4149ac0 2024-03-29T11:42:02.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.127+0000 7f2cb67fc640 1 -- 172.21.15.22:0/3432854503 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f2cb8052770 con 0x7f2cc4149ac0 2024-03-29T11:42:02.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.127+0000 7f2cb67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.127+0000 7f2cb67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:02.131 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:42:02.131 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:02.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.131+0000 7f2ccbc0e640 10 librados: watch_flush enter 2024-03-29T11:42:02.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.131+0000 7f2ccbc0e640 10 librados: watch_flush exit 2024-03-29T11:42:02.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.131+0000 7f2ccbc0e640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T11:42:02.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.131+0000 7f2ccbc0e640 10 client.4523.objecter successfully canceled tick 2024-03-29T11:42:02.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.133+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 >> v1:172.21.15.204:6816/3848099522 conn(0x7f2ca806e000 legacy=0x7f2ca80704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.133+0000 7f2ccbc0e640 10 monclient: shutdown 2024-03-29T11:42:02.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.133+0000 7f2ccbc0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:02.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.133+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 >> v1:172.21.15.22:6790/0 conn(0x7f2cc4149ac0 legacy=0x7f2cc413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.136+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 shutdown_connections 2024-03-29T11:42:02.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.136+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 >> 172.21.15.22:0/3432854503 conn(0x7f2cc40b35e0 msgr2=0x7f2cc413dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:02.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.136+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 shutdown_connections 2024-03-29T11:42:02.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.137+0000 7f2ccbc0e640 1 -- 172.21.15.22:0/3432854503 wait complete. 2024-03-29T11:42:02.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:02.137+0000 7f2ccbc0e640 1 librados: shutdown 2024-03-29T11:42:02.156 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:08.962 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.108+0000 7fd6cb584640 10 monclient: get_monmap_and_config 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.108+0000 7fd6cb584640 10 monclient: build_initial_monmap 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.108+0000 7fd6cb584640 10 monclient: monmap: 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:42:09.110052+0000 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:42:09.110052+0000 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:09.112 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:09.113 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:42:09.113 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:42:09.113 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:09.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.110+0000 7fd6cb584640 1 Processor -- start 2024-03-29T11:42:09.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.110+0000 7fd6cb584640 1 -- start start 2024-03-29T11:42:09.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.110+0000 7fd6cb584640 10 monclient: init 2024-03-29T11:42:09.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:09.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:09.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 10 monclient(hunting): picked mon.noname-a con 0x7fd6bc0389e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:09.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 10 monclient(hunting): picked mon.noname-c con 0x7fd6bc038dc0 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 10 monclient(hunting): picked mon.noname-b con 0x7fd6bc030660 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc0c54c0 con 0x7fd6bc0389e0 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc036430 con 0x7fd6bc038dc0 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc039840 con 0x7fd6bc030660 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cb584640 10 monclient(hunting): authenticate will time out at 1379.388675s 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cad83640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fd6bc030660 0x7fd6bc0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52174/0 (socket says 172.21.15.22:52174) 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.111+0000 7fd6cad83640 1 -- 172.21.15.22:0/1900678105 learned_addr learned my addr 172.21.15.22:0/1900678105 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188020536 0 0) 0x7fd6bc039840 con 0x7fd6bc030660 2024-03-29T11:42:09.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b0003520 con 0x7fd6bc030660 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2553663523 0 0) 0x7fd6b0003520 con 0x7fd6bc030660 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 >> v1:172.21.15.204:6789/0 conn(0x7fd6bc038dc0 legacy=0x7fd6bc0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 >> v1:172.21.15.22:6789/0 conn(0x7fd6bc0389e0 legacy=0x7fd6bc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient: _finish_hunting 0 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 1 monclient: found mon.noname-b 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc035690 con 0x7fd6bc030660 2024-03-29T11:42:09.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient: _finish_auth 0 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient: _check_auth_tickets 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd6c0001690 con 0x7fd6bc030660 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6c9580640 10 monclient: dump: 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:09.119 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6cb584640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.112+0000 7fd6cb584640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6c0001860 con 0x7fd6bc030660 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6c9580640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6c9580640 1 -- 172.21.15.22:0/1900678105 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd6c0001cb0 con 0x7fd6bc030660 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6c9580640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6c9580640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6c9580640 10 monclient: dump: 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.120 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.121 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.121 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:09.121 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:09.121 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:09.121 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:09.121 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:09.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6cb584640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:09.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6cb584640 4 set_mon_vals no callback set 2024-03-29T11:42:09.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6cb584640 10 monclient: shutdown 2024-03-29T11:42:09.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6cb584640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:09.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.113+0000 7fd6cb584640 1 -- 172.21.15.22:0/1900678105 >> v1:172.21.15.22:6790/0 conn(0x7fd6bc030660 legacy=0x7fd6bc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.114+0000 7fd6cb584640 1 -- 172.21.15.22:0/1900678105 shutdown_connections 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.114+0000 7fd6cb584640 1 -- 172.21.15.22:0/1900678105 >> 172.21.15.22:0/1900678105 conn(0x7fd6bc005c00 msgr2=0x7fd6bc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.114+0000 7fd6cb584640 1 -- 172.21.15.22:0/1900678105 shutdown_connections 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.114+0000 7fd6cb584640 1 -- 172.21.15.22:0/1900678105 wait complete. 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.115+0000 7fd6cb584640 10 monclient: build_initial_monmap 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.115+0000 7fd6cb584640 10 monclient: monmap: 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:42:09.128 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.116+0000 7fd6cb584640 1 librados: starting msgr at 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.116+0000 7fd6cb584640 1 librados: starting objecter 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.116+0000 7fd6cb584640 1 Processor -- start 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.116+0000 7fd6cb584640 1 -- start start 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.116+0000 7fd6cb584640 1 librados: setting wanted keys 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.116+0000 7fd6cb584640 1 librados: calling monclient init 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.116+0000 7fd6cb584640 10 monclient: init 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 10 monclient(hunting): picked mon.noname-c con 0x7fd6bc030660 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 10 monclient(hunting): picked mon.noname-b con 0x7fd6bc0389e0 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 10 monclient(hunting): picked mon.noname-a con 0x7fd6bc0c7ea0 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:09.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cad83640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fd6bc0c7ea0 0x7fd6bc15cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:47474/0 (socket says 172.21.15.22:47474) 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cad83640 1 -- 172.21.15.22:0/3400519669 learned_addr learned my addr 172.21.15.22:0/3400519669 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc02ea20 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc15ff30 con 0x7fd6bc0389e0 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd6bc0c7ea0 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6bc0c8560 con 0x7fd6bc030660 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.117+0000 7fd6cb584640 10 monclient(hunting): authenticate will time out at 1379.395001s 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd6bc030660 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70848768 0 0) 0x7fd6bc02ea20 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a8004eb0 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325534812 0 0) 0x7fd6bc0c8560 con 0x7fd6bc030660 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6bc02ea20 con 0x7fd6bc030660 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2266127458 0 0) 0x7fd6a8004eb0 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6bc0c8560 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd6c0001b70 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.118+0000 7fd6b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:09.131 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3745557740 0 0) 0x7fd6bc02ea20 con 0x7fd6bc030660 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6a8004eb0 con 0x7fd6bc030660 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd6c404fdd0 con 0x7fd6bc030660 2024-03-29T11:42:09.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3591882558 0 0) 0x7fd6bc0c8560 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 >> v1:172.21.15.22:6790/0 conn(0x7fd6bc030660 legacy=0x7fd6bc160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 >> v1:172.21.15.204:6789/0 conn(0x7fd6bc0389e0 legacy=0x7fd6bc15c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 1 monclient: found mon.a 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc160b60 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 10 monclient: _renew_subs 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc15d630 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 10 monclient: _renew_subs 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc02d7f0 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.119+0000 7fd6cb584640 1 librados: init done 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.120+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6c0004bf0 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.120+0000 7fd6b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.120+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd6c0004f00 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.120+0000 7fd6c8d7f640 4 set_mon_vals no callback set 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.121+0000 7fd6b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.121+0000 7fd6b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.121+0000 7fd6b6ffd640 10 monclient: dump: 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:09.135 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.122+0000 7fd6cb584640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.122+0000 7fd6cb584640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.122+0000 7fd6cb584640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.122+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc007b90 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.122+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fd6c001aed0 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.123+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fd6c0084bf0 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.123+0000 7fd6ca582640 10 monclient: get_auth_request con 0x7fd6a806dcf0 auth_method 0 2024-03-29T11:42:09.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.123+0000 7fd6b6ffd640 10 client.4506.objecter ms_dispatch 0x7fd6c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:09.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.123+0000 7fd6b6ffd640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:09.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.123+0000 7fd6b6ffd640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:09.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.123+0000 7fd6b6ffd640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T11:42:09.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.124+0000 7fd6b6ffd640 10 client.4506.objecter ms_handle_connect 0x7fd6a806dcf0 2024-03-29T11:42:09.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.130+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6c00504d0 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.130+0000 7fd6b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.130+0000 7fd6b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:09.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.459+0000 7fd6cb584640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.459+0000 7fd6cb584640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.459+0000 7fd6cb584640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:09.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.459+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6bc15fd90 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.460+0000 7fd6b6ffd640 1 -- 172.21.15.22:0/3400519669 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fd6bc15fd90 con 0x7fd6bc0c7ea0 2024-03-29T11:42:09.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.460+0000 7fd6b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.460+0000 7fd6b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:09.464 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:42:09.464 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:09.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.463+0000 7fd6cb584640 10 librados: watch_flush enter 2024-03-29T11:42:09.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.463+0000 7fd6cb584640 10 librados: watch_flush exit 2024-03-29T11:42:09.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.463+0000 7fd6cb584640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T11:42:09.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.463+0000 7fd6cb584640 10 client.4506.objecter successfully canceled tick 2024-03-29T11:42:09.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.464+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 >> v1:172.21.15.204:6816/3848099522 conn(0x7fd6a806dcf0 legacy=0x7fd6a80701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.464+0000 7fd6cb584640 10 monclient: shutdown 2024-03-29T11:42:09.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.464+0000 7fd6cb584640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:09.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.464+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 >> v1:172.21.15.22:6789/0 conn(0x7fd6bc0c7ea0 legacy=0x7fd6bc15cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.464+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 shutdown_connections 2024-03-29T11:42:09.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.464+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 >> 172.21.15.22:0/3400519669 conn(0x7fd6bc005c00 msgr2=0x7fd6bc034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:09.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.465+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 shutdown_connections 2024-03-29T11:42:09.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.465+0000 7fd6cb584640 1 -- 172.21.15.22:0/3400519669 wait complete. 2024-03-29T11:42:09.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:09.465+0000 7fd6cb584640 1 librados: shutdown 2024-03-29T11:42:09.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:16.288 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:16.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.435+0000 7f2937fcf640 10 monclient: get_monmap_and_config 2024-03-29T11:42:16.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.435+0000 7f2937fcf640 10 monclient: build_initial_monmap 2024-03-29T11:42:16.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.435+0000 7f2937fcf640 10 monclient: monmap: 2024-03-29T11:42:16.437 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:16.437 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:16.437 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:42:16.435283+0000 2024-03-29T11:42:16.437 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:42:16.435283+0000 2024-03-29T11:42:16.437 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:16.438 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:16.438 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:16.438 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:42:16.438 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:42:16.438 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:16.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.436+0000 7f2937fcf640 1 Processor -- start 2024-03-29T11:42:16.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.436+0000 7f2937fcf640 1 -- start start 2024-03-29T11:42:16.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.436+0000 7f2937fcf640 10 monclient: init 2024-03-29T11:42:16.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:16.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:16.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f293006ee40 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:16.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f29300aeda0 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:16.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f293013c470 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:16.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293006fdb0 con 0x7f293006ee40 2024-03-29T11:42:16.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2930071090 con 0x7f293013c470 2024-03-29T11:42:16.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2930071370 con 0x7f29300aeda0 2024-03-29T11:42:16.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:16.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2937fcf640 10 monclient(hunting): authenticate will time out at 1386.713559s 2024-03-29T11:42:16.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2935d6b640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f293006ee40 0x7f293013bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:56518/0 (socket says 172.21.15.22:56518) 2024-03-29T11:42:16.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.437+0000 7f2935d6b640 1 -- 172.21.15.22:0/2406546019 learned_addr learned my addr 172.21.15.22:0/2406546019 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401434417 0 0) 0x7f2930071370 con 0x7f29300aeda0 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f291c005730 con 0x7f29300aeda0 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246161672 0 0) 0x7f293006fdb0 con 0x7f293006ee40 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.438+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2930071370 con 0x7f293006ee40 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3209056077 0 0) 0x7f291c005730 con 0x7f29300aeda0 2024-03-29T11:42:16.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 >> v1:172.21.15.204:6789/0 conn(0x7f293013c470 legacy=0x7f293014a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 >> v1:172.21.15.22:6789/0 conn(0x7f293006ee40 legacy=0x7f293013bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: _finish_hunting 0 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 1 monclient: found mon.noname-b 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930070220 con 0x7f29300aeda0 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: _finish_auth 0 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: _check_auth_tickets 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2928001a10 con 0x7f29300aeda0 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: dump: 2024-03-29T11:42:16.442 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2937fcf640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:42:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2937fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:16.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2928001b70 con 0x7f29300aeda0 2024-03-29T11:42:16.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:16.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 1 -- 172.21.15.22:0/2406546019 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2928001fc0 con 0x7f29300aeda0 2024-03-29T11:42:16.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:16.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2934d69640 10 monclient: dump: 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2937fcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.439+0000 7f2937fcf640 4 set_mon_vals no callback set 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.440+0000 7f2937fcf640 10 monclient: shutdown 2024-03-29T11:42:16.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.440+0000 7f2937fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.440+0000 7f2937fcf640 1 -- 172.21.15.22:0/2406546019 >> v1:172.21.15.22:6790/0 conn(0x7f29300aeda0 legacy=0x7f2930071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.440+0000 7f2937fcf640 1 -- 172.21.15.22:0/2406546019 shutdown_connections 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.440+0000 7f2937fcf640 1 -- 172.21.15.22:0/2406546019 >> 172.21.15.22:0/2406546019 conn(0x7f2930067470 msgr2=0x7f29300b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.440+0000 7f2937fcf640 1 -- 172.21.15.22:0/2406546019 shutdown_connections 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.441+0000 7f2937fcf640 1 -- 172.21.15.22:0/2406546019 wait complete. 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.441+0000 7f2937fcf640 10 monclient: build_initial_monmap 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.441+0000 7f2937fcf640 10 monclient: monmap: 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:16.447 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.442+0000 7f2937fcf640 1 librados: starting msgr at 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.442+0000 7f2937fcf640 1 librados: starting objecter 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.442+0000 7f2937fcf640 1 Processor -- start 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.442+0000 7f2937fcf640 1 -- start start 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.442+0000 7f2937fcf640 1 librados: setting wanted keys 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.442+0000 7f2937fcf640 1 librados: calling monclient init 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.442+0000 7f2937fcf640 10 monclient: init 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f293006ee40 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f29300aeda0 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:16.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f293013c470 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29300b3080 con 0x7f29300aeda0 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29301dac70 con 0x7f293013c470 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29300b2ba0 con 0x7f293006ee40 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2937fcf640 10 monclient(hunting): authenticate will time out at 1386.719399s 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2935d6b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f293006ee40 0x7f29301da560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47714/0 (socket says 172.21.15.22:47714) 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f2935d6b640 1 -- 172.21.15.22:0/1694670353 learned_addr learned my addr 172.21.15.22:0/1694670353 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 10 client.?.objecter ms_handle_connect 0x7f293006ee40 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 10 client.?.objecter ms_handle_connect 0x7f293013c470 2024-03-29T11:42:16.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320355933 0 0) 0x7f29301dac70 con 0x7f293013c470 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.443+0000 7f29267fc640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2914004eb0 con 0x7f293013c470 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145662732 0 0) 0x7f29300b2ba0 con 0x7f293006ee40 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29301dac70 con 0x7f293006ee40 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 client.?.objecter ms_handle_connect 0x7f29300aeda0 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903523657 0 0) 0x7f29300b3080 con 0x7f29300aeda0 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29300b2ba0 con 0x7f29300aeda0 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 869892791 0 0) 0x7f29301dac70 con 0x7f293006ee40 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29300b3080 con 0x7f293006ee40 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2920001a50 con 0x7f293006ee40 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): dump: 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.444+0000 7f29267fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:16.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2264432480 0 0) 0x7f29300b2ba0 con 0x7f29300aeda0 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29301dac70 con 0x7f29300aeda0 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2928001e80 con 0x7f29300aeda0 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 10 monclient(hunting): dump: 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:16.455 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 19607691 0 0) 0x7f29300b3080 con 0x7f293006ee40 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 >> v1:172.21.15.204:6789/0 conn(0x7f293013c470 legacy=0x7f29301e1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 >> v1:172.21.15.22:6789/0 conn(0x7f29300aeda0 legacy=0x7f29300b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 1 monclient: found mon.c 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300b3510 con 0x7f293006ee40 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f29267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 10 monclient: _renew_subs 2024-03-29T11:42:16.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29300b37e0 con 0x7f293006ee40 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 10 monclient: _renew_subs 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29301cd3e0 con 0x7f293006ee40 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.445+0000 7f2937fcf640 1 librados: init done 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.446+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2920001630 con 0x7f293006ee40 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.446+0000 7f29267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.446+0000 7f29277fe640 4 set_mon_vals no callback set 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.447+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2920004a90 con 0x7f293006ee40 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.447+0000 7f29267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.447+0000 7f29267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.447+0000 7f29267fc640 10 monclient: dump: 2024-03-29T11:42:16.457 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.447+0000 7f2937fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.447+0000 7f2937fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.447+0000 7f2937fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:16.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.447+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2930140460 con 0x7f293006ee40 2024-03-29T11:42:16.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.448+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f292001ab20 con 0x7f293006ee40 2024-03-29T11:42:16.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.448+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f2920085c80 con 0x7f293006ee40 2024-03-29T11:42:16.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.448+0000 7f29267fc640 10 client.4535.objecter ms_dispatch 0x7f2930000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:16.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.448+0000 7f29267fc640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:16.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.448+0000 7f29267fc640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:16.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.449+0000 7f29267fc640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:42:16.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.449+0000 7f293556a640 10 monclient: get_auth_request con 0x7f291406e080 auth_method 0 2024-03-29T11:42:16.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.449+0000 7f29267fc640 10 client.4535.objecter ms_handle_connect 0x7f291406e080 2024-03-29T11:42:16.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.466+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2920050360 con 0x7f293006ee40 2024-03-29T11:42:16.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.466+0000 7f29267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.466+0000 7f29267fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:16.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.848+0000 7f2937fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.848+0000 7f2937fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.848+0000 7f2937fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:16.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.848+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2930075f10 con 0x7f293006ee40 2024-03-29T11:42:16.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.849+0000 7f29267fc640 1 -- 172.21.15.22:0/1694670353 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f2920050360 con 0x7f293006ee40 2024-03-29T11:42:16.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.849+0000 7f29267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.849+0000 7f29267fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:16.851 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:42:16.852 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:16.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.852+0000 7f2937fcf640 10 librados: watch_flush enter 2024-03-29T11:42:16.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.852+0000 7f2937fcf640 10 librados: watch_flush exit 2024-03-29T11:42:16.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.852+0000 7f2937fcf640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:42:16.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.852+0000 7f2937fcf640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:42:16.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.852+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 >> v1:172.21.15.204:6816/3848099522 conn(0x7f291406e080 legacy=0x7f2914070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.852+0000 7f2937fcf640 10 monclient: shutdown 2024-03-29T11:42:16.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.853+0000 7f2937fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:16.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.853+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 >> v1:172.21.15.22:6790/0 conn(0x7f293006ee40 legacy=0x7f29301da560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.853+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 shutdown_connections 2024-03-29T11:42:16.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.853+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 >> 172.21.15.22:0/1694670353 conn(0x7f2930067470 msgr2=0x7f293006b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:16.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.853+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 shutdown_connections 2024-03-29T11:42:16.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.853+0000 7f2937fcf640 1 -- 172.21.15.22:0/1694670353 wait complete. 2024-03-29T11:42:16.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:16.853+0000 7f2937fcf640 1 librados: shutdown 2024-03-29T11:42:16.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:23.677 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:23.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.809+0000 7fa475315640 10 monclient: get_monmap_and_config 2024-03-29T11:42:23.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.809+0000 7fa475315640 10 monclient: build_initial_monmap 2024-03-29T11:42:23.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.809+0000 7fa475315640 10 monclient: monmap: 2024-03-29T11:42:23.814 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:23.814 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:23.814 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:42:23.809928+0000 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:42:23.809928+0000 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.810+0000 7fa475315640 1 Processor -- start 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.810+0000 7fa475315640 1 -- start start 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.810+0000 7fa475315640 10 monclient: init 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.812+0000 7fa475315640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.812+0000 7fa475315640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.812+0000 7fa475315640 10 monclient(hunting): picked mon.noname-c con 0x7fa4700af6c0 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:23.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.812+0000 7fa475315640 10 monclient(hunting): picked mon.noname-b con 0x7fa47014f6b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.812+0000 7fa46effd640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa4700af6c0 0x7fa4700ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:43514/0 (socket says 172.21.15.22:43514) 2024-03-29T11:42:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.812+0000 7fa46e7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa47014f6b0 0x7fa47014fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50206/0 (socket says 172.21.15.22:50206) 2024-03-29T11:42:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa475315640 10 monclient(hunting): picked mon.noname-a con 0x7fa470152980 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa46effd640 1 -- 172.21.15.22:0/2641577761 learned_addr learned my addr 172.21.15.22:0/2641577761 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa475315640 1 -- 172.21.15.22:0/2641577761 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4700a94e0 con 0x7fa470152980 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa475315640 1 -- 172.21.15.22:0/2641577761 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4700afb60 con 0x7fa4700af6c0 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa475315640 1 -- 172.21.15.22:0/2641577761 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4700aab80 con 0x7fa47014f6b0 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa475315640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa475315640 10 monclient(hunting): authenticate will time out at 1394.089217s 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046997792 0 0) 0x7fa4700aab80 con 0x7fa47014f6b0 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa46dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa46dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa46dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa46dffb640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:42:23.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.813+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa458005730 con 0x7fa47014f6b0 2024-03-29T11:42:23.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.814+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127230483 0 0) 0x7fa458005730 con 0x7fa47014f6b0 2024-03-29T11:42:23.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.814+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 >> v1:172.21.15.204:6789/0 conn(0x7fa4700af6c0 legacy=0x7fa4700ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.814+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 >> v1:172.21.15.22:6789/0 conn(0x7fa470152980 legacy=0x7fa470152e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:23.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:23.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 1 monclient: found mon.noname-b 2024-03-29T11:42:23.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4700aa680 con 0x7fa47014f6b0 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa464001690 con 0x7fa47014f6b0 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: dump: 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:23.820 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa475315640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa475315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa464001860 con 0x7fa47014f6b0 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 1 -- 172.21.15.22:0/2641577761 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa464001cb0 con 0x7fa47014f6b0 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa46dffb640 10 monclient: dump: 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:23.822 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:23.823 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:23.823 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa475315640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.815+0000 7fa475315640 4 set_mon_vals no callback set 2024-03-29T11:42:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.816+0000 7fa475315640 10 monclient: shutdown 2024-03-29T11:42:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.816+0000 7fa475315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.816+0000 7fa475315640 1 -- 172.21.15.22:0/2641577761 >> v1:172.21.15.22:6790/0 conn(0x7fa47014f6b0 legacy=0x7fa47014fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.818+0000 7fa475315640 1 -- 172.21.15.22:0/2641577761 shutdown_connections 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.818+0000 7fa475315640 1 -- 172.21.15.22:0/2641577761 >> 172.21.15.22:0/2641577761 conn(0x7fa4700b35e0 msgr2=0x7fa4700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.818+0000 7fa475315640 1 -- 172.21.15.22:0/2641577761 shutdown_connections 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.818+0000 7fa475315640 1 -- 172.21.15.22:0/2641577761 wait complete. 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.819+0000 7fa475315640 10 monclient: build_initial_monmap 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.819+0000 7fa475315640 10 monclient: monmap: 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:23.829 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.819+0000 7fa475315640 1 librados: starting msgr at 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.819+0000 7fa475315640 1 librados: starting objecter 2024-03-29T11:42:23.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.819+0000 7fa475315640 1 Processor -- start 2024-03-29T11:42:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.820+0000 7fa475315640 1 -- start start 2024-03-29T11:42:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.820+0000 7fa475315640 1 librados: setting wanted keys 2024-03-29T11:42:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.820+0000 7fa475315640 1 librados: calling monclient init 2024-03-29T11:42:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.820+0000 7fa475315640 10 monclient: init 2024-03-29T11:42:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa475315640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa475315640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa475315640 10 monclient(hunting): picked mon.noname-a con 0x7fa4700af6c0 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa475315640 10 monclient(hunting): picked mon.noname-c con 0x7fa47014f6b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa475315640 10 monclient(hunting): picked mon.noname-b con 0x7fa470152980 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa475315640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4701effd0 con 0x7fa4700af6c0 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa475315640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa470086b50 con 0x7fa470152980 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa46f7fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa470152980 0x7fa4700850d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:46390/0 (socket says 172.21.15.22:46390) 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.821+0000 7fa46f7fe640 1 -- 172.21.15.22:0/3604877903 learned_addr learned my addr 172.21.15.22:0/3604877903 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4701f0b70 con 0x7fa47014f6b0 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa475315640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa475315640 10 monclient(hunting): authenticate will time out at 1394.098065s 2024-03-29T11:42:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 client.?.objecter ms_handle_connect 0x7fa470152980 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 client.?.objecter ms_handle_connect 0x7fa47014f6b0 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303932612 0 0) 0x7fa470086b50 con 0x7fa470152980 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:42:23.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa45c004eb0 con 0x7fa470152980 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102530440 0 0) 0x7fa4701f0b70 con 0x7fa47014f6b0 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa470086b50 con 0x7fa47014f6b0 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3300826099 0 0) 0x7fa45c004eb0 con 0x7fa470152980 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4701f0b70 con 0x7fa470152980 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa468001a50 con 0x7fa470152980 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:23.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): dump: 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.822+0000 7fa4577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.823+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1592816675 0 0) 0x7fa4701f0b70 con 0x7fa470152980 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.823+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 >> v1:172.21.15.22:6790/0 conn(0x7fa47014f6b0 legacy=0x7fa47014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.823+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 >> v1:172.21.15.22:6789/0 conn(0x7fa4700af6c0 legacy=0x7fa4701f0460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 1 monclient: found mon.b 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4701f0d00 con 0x7fa470152980 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 10 monclient: _renew_subs 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:42:23.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa470070ee0 con 0x7fa470152980 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 10 monclient: _renew_subs 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47014f020 con 0x7fa470152980 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa475315640 1 librados: init done 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa468001f60 con 0x7fa470152980 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fa468004d80 con 0x7fa470152980 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.824+0000 7fa4577fe640 10 monclient: dump: 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:23.837 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.825+0000 7fa46d7fa640 4 set_mon_vals no callback set 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.825+0000 7fa475315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.826+0000 7fa475315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.826+0000 7fa475315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.826+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa450005080 con 0x7fa470152980 2024-03-29T11:42:23.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.826+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fa46801ad50 con 0x7fa470152980 2024-03-29T11:42:23.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.827+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fa468084d20 con 0x7fa470152980 2024-03-29T11:42:23.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.827+0000 7fa4577fe640 10 client.4471.objecter ms_dispatch 0x7fa470000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.827+0000 7fa4577fe640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.827+0000 7fa4577fe640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.828+0000 7fa4577fe640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-29T11:42:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.829+0000 7fa46effd640 10 monclient: get_auth_request con 0x7fa45c06dc10 auth_method 0 2024-03-29T11:42:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.830+0000 7fa4577fe640 10 client.4471.objecter ms_handle_connect 0x7fa45c06dc10 2024-03-29T11:42:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.835+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa468050600 con 0x7fa470152980 2024-03-29T11:42:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.835+0000 7fa4577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:23.835+0000 7fa4577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:24.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.144+0000 7fa475315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.144+0000 7fa475315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.144+0000 7fa475315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:42:24.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.144+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa450005330 con 0x7fa470152980 2024-03-29T11:42:24.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.145+0000 7fa4577fe640 1 -- 172.21.15.22:0/3604877903 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fa468052530 con 0x7fa470152980 2024-03-29T11:42:24.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.145+0000 7fa4577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.145+0000 7fa4577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:24.147 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:42:24.147 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:24.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.147+0000 7fa475315640 10 librados: watch_flush enter 2024-03-29T11:42:24.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.147+0000 7fa475315640 10 librados: watch_flush exit 2024-03-29T11:42:24.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.147+0000 7fa475315640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-29T11:42:24.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.147+0000 7fa475315640 10 client.4471.objecter successfully canceled tick 2024-03-29T11:42:24.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.147+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 >> v1:172.21.15.204:6816/3848099522 conn(0x7fa45c06dc10 legacy=0x7fa45c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.147+0000 7fa475315640 10 monclient: shutdown 2024-03-29T11:42:24.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.147+0000 7fa475315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:24.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.147+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 >> v1:172.21.15.204:6789/0 conn(0x7fa470152980 legacy=0x7fa4700850d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.148+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 shutdown_connections 2024-03-29T11:42:24.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.148+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 >> 172.21.15.22:0/3604877903 conn(0x7fa4700b35e0 msgr2=0x7fa4700b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:24.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.148+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 shutdown_connections 2024-03-29T11:42:24.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.148+0000 7fa475315640 1 -- 172.21.15.22:0/3604877903 wait complete. 2024-03-29T11:42:24.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:24.148+0000 7fa475315640 1 librados: shutdown 2024-03-29T11:42:24.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:30.972 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:31.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.112+0000 7f5fb2fd6640 10 monclient: get_monmap_and_config 2024-03-29T11:42:31.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.112+0000 7f5fb2fd6640 10 monclient: build_initial_monmap 2024-03-29T11:42:31.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.112+0000 7f5fb2fd6640 10 monclient: monmap: 2024-03-29T11:42:31.116 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:31.117 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:31.117 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:42:31.113866+0000 2024-03-29T11:42:31.118 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:42:31.113866+0000 2024-03-29T11:42:31.118 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:31.118 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:31.118 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:31.118 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:42:31.118 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:42:31.118 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:31.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.113+0000 7f5fb2fd6640 1 Processor -- start 2024-03-29T11:42:31.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.113+0000 7f5fb2fd6640 1 -- start start 2024-03-29T11:42:31.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.113+0000 7f5fb2fd6640 10 monclient: init 2024-03-29T11:42:31.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.114+0000 7f5fb2fd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:31.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.114+0000 7f5fb2fd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:31.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.114+0000 7f5fb2fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f5fac0b02e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:31.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.114+0000 7f5fb2fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f5fac0aeb90 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:31.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.115+0000 7f5fb2fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f5fac0aeff0 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:31.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.115+0000 7f5fb2fd6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac13fb10 con 0x7f5fac0b02e0 2024-03-29T11:42:31.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.115+0000 7f5fb2fd6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac0ae280 con 0x7f5fac0aeff0 2024-03-29T11:42:31.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.115+0000 7f5fb2fd6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac1421e0 con 0x7f5fac0aeb90 2024-03-29T11:42:31.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.115+0000 7f5fb2fd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:31.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.115+0000 7f5fb2fd6640 10 monclient(hunting): authenticate will time out at 1401.392067s 2024-03-29T11:42:31.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.117+0000 7f5fabfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5fac0aeb90 0x7f5fac142700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47968/0 (socket says 172.21.15.22:47968) 2024-03-29T11:42:31.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.117+0000 7f5fabfff640 1 -- 172.21.15.22:0/485444458 learned_addr learned my addr 172.21.15.22:0/485444458 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:31.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.118+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036869523 0 0) 0x7f5fac0ae280 con 0x7f5fac0aeff0 2024-03-29T11:42:31.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.118+0000 7f5fab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.118+0000 7f5fab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.118+0000 7f5fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:42:31.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f98003520 con 0x7f5fac0aeff0 2024-03-29T11:42:31.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704735987 0 0) 0x7f5fac13fb10 con 0x7f5fac0b02e0 2024-03-29T11:42:31.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:42:31.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac0ae280 con 0x7f5fac0b02e0 2024-03-29T11:42:31.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475174656 0 0) 0x7f5fac0ae280 con 0x7f5fac0b02e0 2024-03-29T11:42:31.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 >> v1:172.21.15.22:6790/0 conn(0x7f5fac0aeb90 legacy=0x7f5fac142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:31.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 >> v1:172.21.15.204:6789/0 conn(0x7f5fac0aeff0 legacy=0x7f5fac14da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:31.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 1 monclient: found mon.noname-a 2024-03-29T11:42:31.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac145660 con 0x7f5fac0b02e0 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fb2fd6640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.119+0000 7f5fb2fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5f9c001670 con 0x7f5fac0b02e0 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fab7fe640 10 monclient: dump: 2024-03-29T11:42:31.125 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fb2fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fab7fe640 1 -- 172.21.15.22:0/485444458 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f9c001890 con 0x7f5fac0b02e0 2024-03-29T11:42:31.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:31.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fb2fd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:31.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.120+0000 7f5fb2fd6640 4 set_mon_vals no callback set 2024-03-29T11:42:31.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.121+0000 7f5fb2fd6640 10 monclient: shutdown 2024-03-29T11:42:31.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.121+0000 7f5fb2fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:31.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.121+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/485444458 >> v1:172.21.15.22:6789/0 conn(0x7f5fac0b02e0 legacy=0x7f5fac13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.121+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/485444458 shutdown_connections 2024-03-29T11:42:31.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.121+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/485444458 >> 172.21.15.22:0/485444458 conn(0x7f5fac06b660 msgr2=0x7f5fac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:31.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.121+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/485444458 shutdown_connections 2024-03-29T11:42:31.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.121+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/485444458 wait complete. 2024-03-29T11:42:31.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.122+0000 7f5fb2fd6640 10 monclient: build_initial_monmap 2024-03-29T11:42:31.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.122+0000 7f5fb2fd6640 10 monclient: monmap: 2024-03-29T11:42:31.128 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.122+0000 7f5fb2fd6640 1 librados: starting msgr at 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.122+0000 7f5fb2fd6640 1 librados: starting objecter 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.123+0000 7f5fb2fd6640 1 Processor -- start 2024-03-29T11:42:31.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.125+0000 7f5fb2fd6640 1 -- start start 2024-03-29T11:42:31.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.125+0000 7f5fb2fd6640 1 librados: setting wanted keys 2024-03-29T11:42:31.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.125+0000 7f5fb2fd6640 1 librados: calling monclient init 2024-03-29T11:42:31.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.125+0000 7f5fb2fd6640 10 monclient: init 2024-03-29T11:42:31.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.125+0000 7f5fb2fd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:31.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.125+0000 7f5fb2fd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:31.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.125+0000 7f5fb2fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f5fac0aeb90 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:31.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.126+0000 7f5fb2fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f5fac0aeff0 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:31.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.127+0000 7f5fb2fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f5fac0b02e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:31.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.127+0000 7f5fb2fd6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac134460 con 0x7f5fac0aeb90 2024-03-29T11:42:31.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.127+0000 7f5fb2fd6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac1346d0 con 0x7f5fac0aeff0 2024-03-29T11:42:31.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.127+0000 7f5fb2fd6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac1349b0 con 0x7f5fac0b02e0 2024-03-29T11:42:31.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.127+0000 7f5fb2fd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:31.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.127+0000 7f5fb2fd6640 10 monclient(hunting): authenticate will time out at 1401.404112s 2024-03-29T11:42:31.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.128+0000 7f5fb1573640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5fac0b02e0 0x7f5fac1e2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41894/0 (socket says 172.21.15.22:41894) 2024-03-29T11:42:31.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.128+0000 7f5fb1573640 1 -- 172.21.15.22:0/4036697439 learned_addr learned my addr 172.21.15.22:0/4036697439 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:31.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 client.?.objecter ms_handle_connect 0x7f5fac0b02e0 2024-03-29T11:42:31.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065330276 0 0) 0x7f5fac1349b0 con 0x7f5fac0b02e0 2024-03-29T11:42:31.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f8c004eb0 con 0x7f5fac0b02e0 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 client.?.objecter ms_handle_connect 0x7f5fac0aeff0 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 595393206 0 0) 0x7f5f8c004eb0 con 0x7f5fac0b02e0 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fac1349b0 con 0x7f5fac0b02e0 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5fa4001df0 con 0x7f5fac0b02e0 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:31.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.130+0000 7f5fa97fa640 10 monclient(hunting): dump: 2024-03-29T11:42:31.140 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:31.140 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:31.140 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:31.140 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:31.140 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.140 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030099184 0 0) 0x7f5fac1346d0 con 0x7f5fac0aeff0 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f8c004eb0 con 0x7f5fac0aeff0 2024-03-29T11:42:31.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 10 client.?.objecter ms_handle_connect 0x7f5fac0aeb90 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.132+0000 7f5fa97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816400217 0 0) 0x7f5fac134460 con 0x7f5fac0aeb90 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac1346d0 con 0x7f5fac0aeb90 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1866507780 0 0) 0x7f5fac1346d0 con 0x7f5fac0aeb90 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fac134460 con 0x7f5fac0aeb90 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5f9c000fe0 con 0x7f5fac0aeb90 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.133+0000 7f5fa97fa640 10 monclient(hunting): dump: 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2689073782 0 0) 0x7f5fac134460 con 0x7f5fac0aeb90 2024-03-29T11:42:31.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 >> v1:172.21.15.22:6790/0 conn(0x7f5fac0b02e0 legacy=0x7f5fac1e2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 >> v1:172.21.15.204:6789/0 conn(0x7f5fac0aeff0 legacy=0x7f5fac1df040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 1 monclient: found mon.a 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac1322e0 con 0x7f5fac0aeb90 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fa97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fb2fd6640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fb2fd6640 10 monclient: _renew_subs 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fb2fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:31.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac0b3120 con 0x7f5fac0aeb90 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fb2fd6640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fb2fd6640 10 monclient: _renew_subs 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.134+0000 7f5fb2fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.135+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fac135fc0 con 0x7f5fac0aeb90 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.135+0000 7f5fb2fd6640 1 librados: init done 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.136+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f9c001a00 con 0x7f5fac0aeb90 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.136+0000 7f5fa97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.136+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5f9c003be0 con 0x7f5fac0aeb90 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.136+0000 7f5fa97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.136+0000 7f5fa97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.136+0000 7f5fa97fa640 10 monclient: dump: 2024-03-29T11:42:31.145 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.136+0000 7f5faaffd640 4 set_mon_vals no callback set 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.136+0000 7f5fb2fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.137+0000 7f5fb2fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.137+0000 7f5fb2fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:31.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.137+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f90005080 con 0x7f5fac0aeb90 2024-03-29T11:42:31.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.139+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f5f9c003d40 con 0x7f5fac0aeb90 2024-03-29T11:42:31.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.139+0000 7f5fabfff640 10 monclient: get_auth_request con 0x7f5f8c06df90 auth_method 0 2024-03-29T11:42:31.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.139+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f5f9c053630 con 0x7f5fac0aeb90 2024-03-29T11:42:31.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.139+0000 7f5fa97fa640 10 client.4518.objecter ms_dispatch 0x7f5fac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:31.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.139+0000 7f5fa97fa640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:31.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.139+0000 7f5fa97fa640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:31.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.139+0000 7f5fa97fa640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T11:42:31.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.140+0000 7f5fa97fa640 10 client.4518.objecter ms_handle_connect 0x7f5f8c06df90 2024-03-29T11:42:31.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.148+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f9c04f8d0 con 0x7f5fac0aeb90 2024-03-29T11:42:31.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.148+0000 7f5fa97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.148+0000 7f5fa97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:31.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.476+0000 7f5fb2fd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.476+0000 7f5fb2fd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.476+0000 7f5fb2fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:31.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.476+0000 7f5fb2fd6640 1 -- 172.21.15.22:0/4036697439 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f90005330 con 0x7f5fac0aeb90 2024-03-29T11:42:31.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.481+0000 7f5fa97fa640 1 -- 172.21.15.22:0/4036697439 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f5f9c04fa70 con 0x7f5fac0aeb90 2024-03-29T11:42:31.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.481+0000 7f5fa97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.481+0000 7f5fa97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:31.485 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:42:31.485 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:31.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.485+0000 7f5fa2ffd640 10 librados: watch_flush enter 2024-03-29T11:42:31.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.485+0000 7f5fa2ffd640 10 librados: watch_flush exit 2024-03-29T11:42:31.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.485+0000 7f5fa2ffd640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T11:42:31.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.485+0000 7f5fa2ffd640 10 client.4518.objecter successfully canceled tick 2024-03-29T11:42:31.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.485+0000 7f5fa2ffd640 1 -- 172.21.15.22:0/4036697439 >> v1:172.21.15.204:6816/3848099522 conn(0x7f5f8c06df90 legacy=0x7f5f8c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.485+0000 7f5fa2ffd640 10 monclient: shutdown 2024-03-29T11:42:31.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.485+0000 7f5fa2ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:31.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.485+0000 7f5fa2ffd640 1 -- 172.21.15.22:0/4036697439 >> v1:172.21.15.22:6789/0 conn(0x7f5fac0aeb90 legacy=0x7f5fac0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.487+0000 7f5fa2ffd640 1 -- 172.21.15.22:0/4036697439 shutdown_connections 2024-03-29T11:42:31.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.487+0000 7f5fa2ffd640 1 -- 172.21.15.22:0/4036697439 >> 172.21.15.22:0/4036697439 conn(0x7f5fac06b660 msgr2=0x7f5fac138fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:31.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.487+0000 7f5fa2ffd640 1 -- 172.21.15.22:0/4036697439 shutdown_connections 2024-03-29T11:42:31.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.487+0000 7f5fa2ffd640 1 -- 172.21.15.22:0/4036697439 wait complete. 2024-03-29T11:42:31.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:31.487+0000 7f5fa2ffd640 1 librados: shutdown 2024-03-29T11:42:31.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:38.316 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:38.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.467+0000 7f4f6e33b640 10 monclient: get_monmap_and_config 2024-03-29T11:42:38.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.467+0000 7f4f6e33b640 10 monclient: build_initial_monmap 2024-03-29T11:42:38.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.467+0000 7f4f6e33b640 10 monclient: monmap: 2024-03-29T11:42:38.470 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:38.470 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:38.470 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:42:38.468309+0000 2024-03-29T11:42:38.471 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:42:38.468309+0000 2024-03-29T11:42:38.471 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:38.471 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:38.471 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:38.471 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:42:38.471 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:42:38.471 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:38.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.468+0000 7f4f6e33b640 1 Processor -- start 2024-03-29T11:42:38.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.468+0000 7f4f6e33b640 1 -- start start 2024-03-29T11:42:38.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.468+0000 7f4f6e33b640 10 monclient: init 2024-03-29T11:42:38.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:38.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:38.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 10 monclient(hunting): picked mon.noname-a con 0x7f4f681457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 10 monclient(hunting): picked mon.noname-c con 0x7f4f6813f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 10 monclient(hunting): picked mon.noname-b con 0x7f4f680a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680aaef0 con 0x7f4f681457f0 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f68145bd0 con 0x7f4f6813f650 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f68145eb0 con 0x7f4f680a9d70 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.469+0000 7f4f6e33b640 10 monclient(hunting): authenticate will time out at 1408.746906s 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.470+0000 7f4f6c8d8640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4f680a9d70 0x7f4f6814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52038/0 (socket says 172.21.15.22:52038) 2024-03-29T11:42:38.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.470+0000 7f4f6c8d8640 1 -- 172.21.15.22:0/3265314968 learned_addr learned my addr 172.21.15.22:0/3265314968 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:38.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.471+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132675313 0 0) 0x7f4f68145eb0 con 0x7f4f680a9d70 2024-03-29T11:42:38.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.471+0000 7f4f66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.471+0000 7f4f66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.471+0000 7f4f66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.471+0000 7f4f66ffd640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:42:38.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.471+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f50003520 con 0x7f4f680a9d70 2024-03-29T11:42:38.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.472+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2053100770 0 0) 0x7f4f50003520 con 0x7f4f680a9d70 2024-03-29T11:42:38.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.472+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 >> v1:172.21.15.204:6789/0 conn(0x7f4f6813f650 legacy=0x7f4f680b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:38.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.472+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 >> v1:172.21.15.22:6789/0 conn(0x7f4f681457f0 legacy=0x7f4f6813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:38.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:38.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 1 monclient: found mon.noname-b 2024-03-29T11:42:38.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:38.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:42:38.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68146340 con 0x7f4f680a9d70 2024-03-29T11:42:38.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:38.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:38.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f6e33b640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f6e33b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f5c001690 con 0x7f4f680a9d70 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.475+0000 7f4f66ffd640 10 monclient: dump: 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:38.480 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:38.481 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:38.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.476+0000 7f4f6e33b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:38.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.478+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f5c001920 con 0x7f4f680a9d70 2024-03-29T11:42:38.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.478+0000 7f4f66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:38.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.478+0000 7f4f6e33b640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:38.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.478+0000 7f4f6e33b640 4 set_mon_vals no callback set 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.478+0000 7f4f66ffd640 1 -- 172.21.15.22:0/3265314968 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f5c001cb0 con 0x7f4f680a9d70 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.478+0000 7f4f66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.478+0000 7f4f66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.479+0000 7f4f66ffd640 10 monclient: dump: 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.482 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:38.483 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:38.483 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:38.483 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:38.483 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:38.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.479+0000 7f4f6e33b640 10 monclient: shutdown 2024-03-29T11:42:38.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.479+0000 7f4f6e33b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:38.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.479+0000 7f4f6e33b640 1 -- 172.21.15.22:0/3265314968 >> v1:172.21.15.22:6790/0 conn(0x7f4f680a9d70 legacy=0x7f4f6814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.480+0000 7f4f6e33b640 1 -- 172.21.15.22:0/3265314968 shutdown_connections 2024-03-29T11:42:38.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.480+0000 7f4f6e33b640 1 -- 172.21.15.22:0/3265314968 >> 172.21.15.22:0/3265314968 conn(0x7f4f68066a00 msgr2=0x7f4f68066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:38.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.480+0000 7f4f6e33b640 1 -- 172.21.15.22:0/3265314968 shutdown_connections 2024-03-29T11:42:38.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.480+0000 7f4f6e33b640 1 -- 172.21.15.22:0/3265314968 wait complete. 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.482+0000 7f4f6e33b640 10 monclient: build_initial_monmap 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.482+0000 7f4f6e33b640 10 monclient: monmap: 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:38.486 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.482+0000 7f4f6e33b640 1 librados: starting msgr at 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.482+0000 7f4f6e33b640 1 librados: starting objecter 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.482+0000 7f4f6e33b640 1 Processor -- start 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.483+0000 7f4f6e33b640 1 -- start start 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.483+0000 7f4f6e33b640 1 librados: setting wanted keys 2024-03-29T11:42:38.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.483+0000 7f4f6e33b640 1 librados: calling monclient init 2024-03-29T11:42:38.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.483+0000 7f4f6e33b640 10 monclient: init 2024-03-29T11:42:38.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:38.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:38.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 10 monclient(hunting): picked mon.noname-b con 0x7f4f680a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:38.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 10 monclient(hunting): picked mon.noname-c con 0x7f4f6813f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:38.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 10 monclient(hunting): picked mon.noname-a con 0x7f4f681457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:38.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680abf50 con 0x7f4f681457f0 2024-03-29T11:42:38.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f68137a20 con 0x7f4f680a9d70 2024-03-29T11:42:38.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680ac230 con 0x7f4f6813f650 2024-03-29T11:42:38.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:38.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f6e33b640 10 monclient(hunting): authenticate will time out at 1408.761244s 2024-03-29T11:42:38.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f677fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4f6813f650 0x7f4f68136c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55826/0 (socket says 172.21.15.22:55826) 2024-03-29T11:42:38.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.484+0000 7f4f677fe640 1 -- 172.21.15.22:0/326945947 learned_addr learned my addr 172.21.15.22:0/326945947 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:38.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.487+0000 7f4f64ff9640 10 client.?.objecter ms_handle_connect 0x7f4f6813f650 2024-03-29T11:42:38.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.487+0000 7f4f64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.487+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952365546 0 0) 0x7f4f680ac230 con 0x7f4f6813f650 2024-03-29T11:42:38.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.487+0000 7f4f64ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.487+0000 7f4f64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.487+0000 7f4f64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.487+0000 7f4f64ff9640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.487+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f54004eb0 con 0x7f4f6813f650 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 client.?.objecter ms_handle_connect 0x7f4f681457f0 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3582744898 0 0) 0x7f4f54004eb0 con 0x7f4f6813f650 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f680ac230 con 0x7f4f6813f650 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f60001bf0 con 0x7f4f6813f650 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:38.505 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485556423 0 0) 0x7f4f680abf50 con 0x7f4f681457f0 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f54004eb0 con 0x7f4f681457f0 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.488+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2852404789 0 0) 0x7f4f54004eb0 con 0x7f4f681457f0 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.489+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f680abf50 con 0x7f4f681457f0 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.489+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f5c001550 con 0x7f4f681457f0 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.489+0000 7f4f64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.489+0000 7f4f64ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.489+0000 7f4f64ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.507 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.489+0000 7f4f64ff9640 10 client.?.objecter ms_handle_connect 0x7f4f680a9d70 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.489+0000 7f4f64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1463874795 0 0) 0x7f4f680ac230 con 0x7f4f6813f650 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 >> v1:172.21.15.204:6789/0 conn(0x7f4f680a9d70 legacy=0x7f4f68142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 >> v1:172.21.15.22:6789/0 conn(0x7f4f681457f0 legacy=0x7f4f68137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 1 monclient: found mon.c 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:38.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f680ac510 con 0x7f4f6813f650 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f60002180 con 0x7f4f6813f650 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f6e33b640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f6e33b640 10 monclient: _renew_subs 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f6e33b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f680ac7f0 con 0x7f4f6813f650 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.490+0000 7f4f6e33b640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.491+0000 7f4f6e33b640 10 monclient: _renew_subs 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.491+0000 7f4f6e33b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.491+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6813fda0 con 0x7f4f6813f650 2024-03-29T11:42:38.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.491+0000 7f4f6e33b640 1 librados: init done 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.491+0000 7f4f64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.491+0000 7f4f667fc640 4 set_mon_vals no callback set 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.492+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f4f60004e20 con 0x7f4f6813f650 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.492+0000 7f4f64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.492+0000 7f4f64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.492+0000 7f4f64ff9640 10 monclient: dump: 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:38.510 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.493+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f4f6001adf0 con 0x7f4f6813f650 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.493+0000 7f4f6e33b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.493+0000 7f4f6e33b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.493+0000 7f4f6e33b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.493+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f48005080 con 0x7f4f6813f650 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.494+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f4f60084e00 con 0x7f4f6813f650 2024-03-29T11:42:38.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.494+0000 7f4f64ff9640 10 client.4550.objecter ms_dispatch 0x7f4f68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.494+0000 7f4f67fff640 10 monclient: get_auth_request con 0x7f4f5406dfc0 auth_method 0 2024-03-29T11:42:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.494+0000 7f4f64ff9640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.494+0000 7f4f64ff9640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.494+0000 7f4f64ff9640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:42:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.495+0000 7f4f64ff9640 10 client.4550.objecter ms_handle_connect 0x7f4f5406dfc0 2024-03-29T11:42:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.506+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f600506e0 con 0x7f4f6813f650 2024-03-29T11:42:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.506+0000 7f4f64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.506+0000 7f4f64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:38.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.841+0000 7f4f6e33b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:38.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.841+0000 7f4f6e33b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:38.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.841+0000 7f4f6e33b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:38.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.841+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f48005330 con 0x7f4f6813f650 2024-03-29T11:42:38.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.842+0000 7f4f64ff9640 1 -- 172.21.15.22:0/326945947 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f4f60052610 con 0x7f4f6813f650 2024-03-29T11:42:38.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.842+0000 7f4f64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:38.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.842+0000 7f4f64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:38.846 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:42:38.847 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:38.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.846+0000 7f4f6e33b640 10 librados: watch_flush enter 2024-03-29T11:42:38.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.847+0000 7f4f6e33b640 10 librados: watch_flush exit 2024-03-29T11:42:38.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.847+0000 7f4f6e33b640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:42:38.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.847+0000 7f4f6e33b640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:42:38.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.847+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 >> v1:172.21.15.204:6816/3848099522 conn(0x7f4f5406dfc0 legacy=0x7f4f54070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.847+0000 7f4f6e33b640 10 monclient: shutdown 2024-03-29T11:42:38.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.847+0000 7f4f6e33b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:38.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.847+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 >> v1:172.21.15.22:6790/0 conn(0x7f4f6813f650 legacy=0x7f4f68136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.848+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 shutdown_connections 2024-03-29T11:42:38.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.848+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 >> 172.21.15.22:0/326945947 conn(0x7f4f68066a00 msgr2=0x7f4f6813aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:38.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.851+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 shutdown_connections 2024-03-29T11:42:38.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.851+0000 7f4f6e33b640 1 -- 172.21.15.22:0/326945947 wait complete. 2024-03-29T11:42:38.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:38.851+0000 7f4f6e33b640 1 librados: shutdown 2024-03-29T11:42:38.871 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:45.678 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:45.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.828+0000 7fc637190640 10 monclient: get_monmap_and_config 2024-03-29T11:42:45.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.828+0000 7fc637190640 10 monclient: build_initial_monmap 2024-03-29T11:42:45.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.828+0000 7fc637190640 10 monclient: monmap: 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:42:45.829475+0000 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:42:45.829475+0000 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:42:45.832 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:45.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.830+0000 7fc637190640 1 Processor -- start 2024-03-29T11:42:45.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.830+0000 7fc637190640 1 -- start start 2024-03-29T11:42:45.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.830+0000 7fc637190640 10 monclient: init 2024-03-29T11:42:45.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.831+0000 7fc637190640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:45.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.831+0000 7fc637190640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:45.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.832+0000 7fc637190640 10 monclient(hunting): picked mon.noname-c con 0x7fc6301457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:45.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.832+0000 7fc637190640 10 monclient(hunting): picked mon.noname-b con 0x7fc63013f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:45.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.832+0000 7fc637190640 10 monclient(hunting): picked mon.noname-a con 0x7fc6300a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:45.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.832+0000 7fc637190640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6300aaef0 con 0x7fc6300a9d70 2024-03-29T11:42:45.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.832+0000 7fc637190640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc630145bd0 con 0x7fc6301457f0 2024-03-29T11:42:45.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.832+0000 7fc637190640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc630145eb0 con 0x7fc63013f650 2024-03-29T11:42:45.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.832+0000 7fc637190640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:45.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.832+0000 7fc637190640 10 monclient(hunting): authenticate will time out at 1416.109915s 2024-03-29T11:42:45.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.834+0000 7fc634f2c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc6301457f0 0x7fc63013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:59847/0 (socket says 172.21.15.22:59847) 2024-03-29T11:42:45.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.834+0000 7fc634f2c640 1 -- 172.21.15.22:0/1318948972 learned_addr learned my addr 172.21.15.22:0/1318948972 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:45.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.835+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599992529 0 0) 0x7fc630145eb0 con 0x7fc63013f650 2024-03-29T11:42:45.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.836+0000 7fc62f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:45.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.836+0000 7fc62f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:45.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.836+0000 7fc62f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:45.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.836+0000 7fc62f7fe640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:42:45.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.836+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc620003520 con 0x7fc63013f650 2024-03-29T11:42:45.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233706996 0 0) 0x7fc630145bd0 con 0x7fc6301457f0 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc630145eb0 con 0x7fc6301457f0 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308090960 0 0) 0x7fc6300aaef0 con 0x7fc6300a9d70 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:42:45.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.838+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc630145bd0 con 0x7fc6300a9d70 2024-03-29T11:42:45.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4248380196 0 0) 0x7fc620003520 con 0x7fc63013f650 2024-03-29T11:42:45.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 >> v1:172.21.15.204:6789/0 conn(0x7fc6301457f0 legacy=0x7fc63013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:45.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 >> v1:172.21.15.22:6789/0 conn(0x7fc6300a9d70 legacy=0x7fc63014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:45.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:45.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc630146340 con 0x7fc63013f650 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc61c001690 con 0x7fc63013f650 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.839+0000 7fc62f7fe640 10 monclient: dump: 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:45.846 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.840+0000 7fc637190640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:42:45.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.840+0000 7fc637190640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:45.848 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.842+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc61c0017f0 con 0x7fc63013f650 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.842+0000 7fc62f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.842+0000 7fc62f7fe640 1 -- 172.21.15.22:0/1318948972 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc61c001c40 con 0x7fc63013f650 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.842+0000 7fc637190640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.842+0000 7fc637190640 4 set_mon_vals no callback set 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.842+0000 7fc62f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.842+0000 7fc62f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.842+0000 7fc62f7fe640 10 monclient: dump: 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:45.849 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:45.850 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:45.850 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:45.850 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:45.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.843+0000 7fc637190640 10 monclient: shutdown 2024-03-29T11:42:45.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.843+0000 7fc637190640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:45.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.843+0000 7fc637190640 1 -- 172.21.15.22:0/1318948972 >> v1:172.21.15.22:6790/0 conn(0x7fc63013f650 legacy=0x7fc6300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:45.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.843+0000 7fc637190640 1 -- 172.21.15.22:0/1318948972 shutdown_connections 2024-03-29T11:42:45.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.843+0000 7fc637190640 1 -- 172.21.15.22:0/1318948972 >> 172.21.15.22:0/1318948972 conn(0x7fc630066a00 msgr2=0x7fc630066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:45.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.845+0000 7fc637190640 1 -- 172.21.15.22:0/1318948972 shutdown_connections 2024-03-29T11:42:45.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.846+0000 7fc637190640 1 -- 172.21.15.22:0/1318948972 wait complete. 2024-03-29T11:42:45.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.847+0000 7fc637190640 10 monclient: build_initial_monmap 2024-03-29T11:42:45.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.847+0000 7fc637190640 10 monclient: monmap: 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.847+0000 7fc637190640 1 librados: starting msgr at 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.848+0000 7fc637190640 1 librados: starting objecter 2024-03-29T11:42:45.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.848+0000 7fc637190640 1 Processor -- start 2024-03-29T11:42:45.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.848+0000 7fc637190640 1 -- start start 2024-03-29T11:42:45.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.848+0000 7fc637190640 1 librados: setting wanted keys 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.848+0000 7fc637190640 1 librados: calling monclient init 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.848+0000 7fc637190640 10 monclient: init 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.849+0000 7fc637190640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.849+0000 7fc637190640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.849+0000 7fc637190640 10 monclient(hunting): picked mon.noname-b con 0x7fc6300a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.849+0000 7fc637190640 10 monclient(hunting): picked mon.noname-c con 0x7fc6301457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc62ffff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc6301457f0 0x7fc63006f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:54494/0 (socket says 172.21.15.22:54494) 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc637190640 10 monclient(hunting): picked mon.noname-a con 0x7fc6301dc2b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc62ffff640 1 -- 172.21.15.22:0/3083050401 learned_addr learned my addr 172.21.15.22:0/3083050401 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6300b0f10 con 0x7fc6301dc2b0 2024-03-29T11:42:45.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc630071200 con 0x7fc6300a9d70 2024-03-29T11:42:45.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc62d7fa640 10 client.?.objecter ms_handle_connect 0x7fc6301457f0 2024-03-29T11:42:45.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc62d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:45.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6300714e0 con 0x7fc6301457f0 2024-03-29T11:42:45.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc637190640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:45.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.851+0000 7fc637190640 10 monclient(hunting): authenticate will time out at 1416.128756s 2024-03-29T11:42:45.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.852+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498215099 0 0) 0x7fc6300714e0 con 0x7fc6301457f0 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.852+0000 7fc62d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.852+0000 7fc62d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.852+0000 7fc62d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.852+0000 7fc62d7fa640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.852+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc614004eb0 con 0x7fc6301457f0 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.852+0000 7fc62d7fa640 10 client.?.objecter ms_handle_connect 0x7fc6300a9d70 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.852+0000 7fc62d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403612642 0 0) 0x7fc614004eb0 con 0x7fc6301457f0 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6300714e0 con 0x7fc6301457f0 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc61c001650 con 0x7fc6301457f0 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:45.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027975981 0 0) 0x7fc630071200 con 0x7fc6300a9d70 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:45.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:45.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:45.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:42:45.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc614004eb0 con 0x7fc6300a9d70 2024-03-29T11:42:45.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 931711246 0 0) 0x7fc6300714e0 con 0x7fc6301457f0 2024-03-29T11:42:45.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 >> v1:172.21.15.204:6789/0 conn(0x7fc6300a9d70 legacy=0x7fc63006ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:45.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.853+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 >> v1:172.21.15.22:6789/0 conn(0x7fc6301dc2b0 legacy=0x7fc6301e6f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:45.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:45.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 1 monclient: found mon.c 2024-03-29T11:42:45.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6301dbd90 con 0x7fc6301457f0 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 10 monclient: _renew_subs 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63006f520 con 0x7fc6301457f0 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 10 monclient: _renew_subs 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc630071e50 con 0x7fc6301457f0 2024-03-29T11:42:45.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc637190640 1 librados: init done 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc61c0011e0 con 0x7fc6301457f0 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.856+0000 7fc62effd640 4 set_mon_vals no callback set 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.857+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc61c003ae0 con 0x7fc6301457f0 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.857+0000 7fc62d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.857+0000 7fc62d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.857+0000 7fc62d7fa640 10 monclient: dump: 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:45.863 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:45.864 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:45.864 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.859+0000 7fc637190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.859+0000 7fc637190640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.859+0000 7fc637190640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.859+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc610005080 con 0x7fc6301457f0 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.860+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fc61c003c40 con 0x7fc6301457f0 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.861+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fc61c084b70 con 0x7fc6301457f0 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.861+0000 7fc62d7fa640 10 client.4556.objecter ms_dispatch 0x7fc630000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.861+0000 7fc62d7fa640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.861+0000 7fc62d7fa640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:45.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.861+0000 7fc62d7fa640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:42:45.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.862+0000 7fc634f2c640 10 monclient: get_auth_request con 0x7fc61406dd00 auth_method 0 2024-03-29T11:42:45.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.863+0000 7fc62d7fa640 10 client.4556.objecter ms_handle_connect 0x7fc61406dd00 2024-03-29T11:42:45.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.873+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc61c050280 con 0x7fc6301457f0 2024-03-29T11:42:45.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.873+0000 7fc62d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:45.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:45.873+0000 7fc62d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:46.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.203+0000 7fc637190640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.203+0000 7fc637190640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.203+0000 7fc637190640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:42:46.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.203+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc610005330 con 0x7fc6301457f0 2024-03-29T11:42:46.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.205+0000 7fc62d7fa640 1 -- 172.21.15.22:0/3083050401 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fc61c052380 con 0x7fc6301457f0 2024-03-29T11:42:46.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.205+0000 7fc62d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.205+0000 7fc62d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:46.210 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:42:46.210 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:46.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.209+0000 7fc637190640 10 librados: watch_flush enter 2024-03-29T11:42:46.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.209+0000 7fc637190640 10 librados: watch_flush exit 2024-03-29T11:42:46.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.209+0000 7fc637190640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:42:46.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.209+0000 7fc637190640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:42:46.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.209+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 >> v1:172.21.15.204:6816/3848099522 conn(0x7fc61406dd00 legacy=0x7fc6140701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.209+0000 7fc637190640 10 monclient: shutdown 2024-03-29T11:42:46.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.209+0000 7fc637190640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:46.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.209+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 >> v1:172.21.15.22:6790/0 conn(0x7fc6301457f0 legacy=0x7fc63006f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.210+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 shutdown_connections 2024-03-29T11:42:46.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.210+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 >> 172.21.15.22:0/3083050401 conn(0x7fc630066a00 msgr2=0x7fc630066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:46.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.210+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 shutdown_connections 2024-03-29T11:42:46.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.210+0000 7fc637190640 1 -- 172.21.15.22:0/3083050401 wait complete. 2024-03-29T11:42:46.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:46.210+0000 7fc637190640 1 librados: shutdown 2024-03-29T11:42:46.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:53.039 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:53.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.193+0000 7f59d0e74640 10 monclient: get_monmap_and_config 2024-03-29T11:42:53.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.194+0000 7f59d0e74640 10 monclient: build_initial_monmap 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.194+0000 7f59d0e74640 10 monclient: monmap: 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:42:53.195277+0000 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:42:53.195277+0000 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:42:53.198 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:53.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.194+0000 7f59d0e74640 1 Processor -- start 2024-03-29T11:42:53.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.194+0000 7f59d0e74640 1 -- start start 2024-03-29T11:42:53.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.194+0000 7f59d0e74640 10 monclient: init 2024-03-29T11:42:53.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:53.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:53.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 10 monclient(hunting): picked mon.noname-b con 0x7f59cc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:53.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 10 monclient(hunting): picked mon.noname-c con 0x7f59cc13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:53.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 10 monclient(hunting): picked mon.noname-a con 0x7f59cc0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:53.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc0aaef0 con 0x7f59cc0a9d70 2024-03-29T11:42:53.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc145bd0 con 0x7f59cc13f650 2024-03-29T11:42:53.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc145eb0 con 0x7f59cc1457f0 2024-03-29T11:42:53.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:53.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.195+0000 7f59d0e74640 10 monclient(hunting): authenticate will time out at 1423.472954s 2024-03-29T11:42:53.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.196+0000 7f59c9d9b640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f59cc13f650 0x7f59cc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:49694/0 (socket says 172.21.15.22:49694) 2024-03-29T11:42:53.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.196+0000 7f59c9d9b640 1 -- 172.21.15.22:0/2598166004 learned_addr learned my addr 172.21.15.22:0/2598166004 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:53.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.196+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968868859 0 0) 0x7f59cc145bd0 con 0x7f59cc13f650 2024-03-29T11:42:53.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.196+0000 7f59c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.196+0000 7f59c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.196+0000 7f59c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.196+0000 7f59c959a640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.196+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b4003520 con 0x7f59cc13f650 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611370439 0 0) 0x7f59cc145eb0 con 0x7f59cc1457f0 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc145bd0 con 0x7f59cc1457f0 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2653881462 0 0) 0x7f59b4003520 con 0x7f59cc13f650 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 >> v1:172.21.15.22:6790/0 conn(0x7f59cc1457f0 legacy=0x7f59cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 >> v1:172.21.15.22:6789/0 conn(0x7f59cc0a9d70 legacy=0x7f59cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 monclient: found mon.noname-c 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:53.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc146340 con 0x7f59cc13f650 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: _finish_auth 0 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59d0e74640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59d0e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f59c00015c0 con 0x7f59cc13f650 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: dump: 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.204 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59d0e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c00017e0 con 0x7f59cc13f650 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59d0e74640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59d0e74640 4 set_mon_vals no callback set 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 1 -- 172.21.15.22:0/2598166004 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f59c0001b70 con 0x7f59cc13f650 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:53.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.197+0000 7f59c959a640 10 monclient: dump: 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.198+0000 7f59d0e74640 10 monclient: shutdown 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.198+0000 7f59d0e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:53.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.198+0000 7f59d0e74640 1 -- 172.21.15.22:0/2598166004 >> v1:172.21.15.204:6789/0 conn(0x7f59cc13f650 legacy=0x7f59cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.198+0000 7f59d0e74640 1 -- 172.21.15.22:0/2598166004 shutdown_connections 2024-03-29T11:42:53.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.198+0000 7f59d0e74640 1 -- 172.21.15.22:0/2598166004 >> 172.21.15.22:0/2598166004 conn(0x7f59cc066a00 msgr2=0x7f59cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:53.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.198+0000 7f59d0e74640 1 -- 172.21.15.22:0/2598166004 shutdown_connections 2024-03-29T11:42:53.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.198+0000 7f59d0e74640 1 -- 172.21.15.22:0/2598166004 wait complete. 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.200+0000 7f59d0e74640 10 monclient: build_initial_monmap 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.200+0000 7f59d0e74640 10 monclient: monmap: 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:53.210 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.200+0000 7f59d0e74640 1 librados: starting msgr at 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.200+0000 7f59d0e74640 1 librados: starting objecter 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.200+0000 7f59d0e74640 1 Processor -- start 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.200+0000 7f59d0e74640 1 -- start start 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.200+0000 7f59d0e74640 1 librados: setting wanted keys 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.200+0000 7f59d0e74640 1 librados: calling monclient init 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.201+0000 7f59d0e74640 10 monclient: init 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 10 monclient(hunting): picked mon.noname-a con 0x7f59cc0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:42:53.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 10 monclient(hunting): picked mon.noname-c con 0x7f59cc13f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 10 monclient(hunting): picked mon.noname-b con 0x7f59cc1457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59c9d9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f59cc13f650 0x7f59cc071a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58082/0 (socket says 172.21.15.22:58082) 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59ca59c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f59cc0a9d70 0x7f59cc139c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:41416/0 (socket says 172.21.15.22:41416) 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59c9d9b640 1 -- 172.21.15.22:0/648080556 learned_addr learned my addr 172.21.15.22:0/648080556 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc142650 con 0x7f59cc0a9d70 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc06ec60 con 0x7f59cc1457f0 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc072170 con 0x7f59cc13f650 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59baffd640 10 client.?.objecter ms_handle_connect 0x7f59cc0a9d70 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59baffd640 10 client.?.objecter ms_handle_connect 0x7f59cc13f650 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.202+0000 7f59d0e74640 10 monclient(hunting): authenticate will time out at 1423.479903s 2024-03-29T11:42:53.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 client.?.objecter ms_handle_connect 0x7f59cc1457f0 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660336535 0 0) 0x7f59cc142650 con 0x7f59cc0a9d70 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac004eb0 con 0x7f59cc0a9d70 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 566033024 0 0) 0x7f59ac004eb0 con 0x7f59cc0a9d70 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59cc142650 con 0x7f59cc0a9d70 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124600404 0 0) 0x7f59cc072170 con 0x7f59cc13f650 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:42:53.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac004eb0 con 0x7f59cc13f650 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f59c40018d0 con 0x7f59cc0a9d70 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): dump: 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:53.214 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134714784 0 0) 0x7f59cc06ec60 con 0x7f59cc1457f0 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.203+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc072170 con 0x7f59cc1457f0 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447998708 0 0) 0x7f59ac004eb0 con 0x7f59cc13f650 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59cc06ec60 con 0x7f59cc13f650 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f59c0001500 con 0x7f59cc13f650 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient(hunting): dump: 2024-03-29T11:42:53.215 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3035171209 0 0) 0x7f59cc142650 con 0x7f59cc0a9d70 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 >> v1:172.21.15.22:6790/0 conn(0x7f59cc13f650 legacy=0x7f59cc071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 >> v1:172.21.15.204:6789/0 conn(0x7f59cc1457f0 legacy=0x7f59cc13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:53.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 monclient: found mon.a 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc142be0 con 0x7f59cc0a9d70 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 10 monclient: _renew_subs 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc142e90 con 0x7f59cc0a9d70 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 10 monclient: _renew_subs 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:53.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc142280 con 0x7f59cc0a9d70 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59d0e74640 1 librados: init done 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c4001a30 con 0x7f59cc0a9d70 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f59c40037d0 con 0x7f59cc0a9d70 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.204+0000 7f59baffd640 10 monclient: dump: 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:42:53.218 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:42:53.219 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:42:53.219 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:42:53.219 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:42:53.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.205+0000 7f59c8d99640 4 set_mon_vals no callback set 2024-03-29T11:42:53.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.205+0000 7f59d0e74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.206+0000 7f59d0e74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.206+0000 7f59d0e74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:53.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.206+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b0005080 con 0x7f59cc0a9d70 2024-03-29T11:42:53.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.207+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f59c4003930 con 0x7f59cc0a9d70 2024-03-29T11:42:53.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.207+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f59c4084950 con 0x7f59cc0a9d70 2024-03-29T11:42:53.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.207+0000 7f59baffd640 10 client.4530.objecter ms_dispatch 0x7f59cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:53.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.207+0000 7f59baffd640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:53.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.207+0000 7f59baffd640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:53.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.208+0000 7f59baffd640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T11:42:53.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.208+0000 7f59c9d9b640 10 monclient: get_auth_request con 0x7f59ac06dfb0 auth_method 0 2024-03-29T11:42:53.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.208+0000 7f59baffd640 10 client.4530.objecter ms_handle_connect 0x7f59ac06dfb0 2024-03-29T11:42:53.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.215+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59c4050290 con 0x7f59cc0a9d70 2024-03-29T11:42:53.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.215+0000 7f59baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.215+0000 7f59baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:53.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.583+0000 7f59d0e74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.583+0000 7f59d0e74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.583+0000 7f59d0e74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:42:53.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.583+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59b0005330 con 0x7f59cc0a9d70 2024-03-29T11:42:53.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.584+0000 7f59baffd640 1 -- 172.21.15.22:0/648080556 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f59c40521c0 con 0x7f59cc0a9d70 2024-03-29T11:42:53.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.584+0000 7f59baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.584+0000 7f59baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:53.588 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:42:53.588 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 10 librados: watch_flush enter 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 10 librados: watch_flush exit 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 10 client.4530.objecter successfully canceled tick 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 >> v1:172.21.15.204:6816/3848099522 conn(0x7f59ac06dfb0 legacy=0x7f59ac070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 10 monclient: shutdown 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 >> v1:172.21.15.22:6789/0 conn(0x7f59cc0a9d70 legacy=0x7f59cc139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 shutdown_connections 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 >> 172.21.15.22:0/648080556 conn(0x7f59cc066a00 msgr2=0x7f59cc13b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 shutdown_connections 2024-03-29T11:42:53.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 1 -- 172.21.15.22:0/648080556 wait complete. 2024-03-29T11:42:53.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:42:53.588+0000 7f59d0e74640 1 librados: shutdown 2024-03-29T11:42:53.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:00.414 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:00.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.565+0000 7fb9fb59e640 10 monclient: get_monmap_and_config 2024-03-29T11:43:00.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.565+0000 7fb9fb59e640 10 monclient: build_initial_monmap 2024-03-29T11:43:00.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.565+0000 7fb9fb59e640 10 monclient: monmap: 2024-03-29T11:43:00.570 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:00.566739+0000 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:00.566739+0000 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:00.571 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:00.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.569+0000 7fb9fb59e640 1 Processor -- start 2024-03-29T11:43:00.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.569+0000 7fb9fb59e640 1 -- start start 2024-03-29T11:43:00.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.569+0000 7fb9fb59e640 10 monclient: init 2024-03-29T11:43:00.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:00.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:00.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f4034850 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:00.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f40c51c0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:00.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f40c7cd0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f4029190 con 0x7fb9f4034850 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40293e0 con 0x7fb9f40c51c0 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40c8340 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.570+0000 7fb9fb59e640 10 monclient(hunting): authenticate will time out at 1430.847549s 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.574+0000 7fb9fad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb9f40c7cd0 0x7fb9f4029f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53815/0 (socket says 172.21.15.22:53815) 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.574+0000 7fb9fad9d640 1 -- 172.21.15.22:0/616228650 learned_addr learned my addr 172.21.15.22:0/616228650 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.577+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321943273 0 0) 0x7fb9f40c8340 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.577+0000 7fb9f959a640 10 monclient(hunting): _negotiate 2024-03-29T11:43:00.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.577+0000 7fb9f959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:00.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.577+0000 7fb9f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:00.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.577+0000 7fb9f959a640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:43:00.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.577+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e4003520 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.578+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2407518303 0 0) 0x7fb9e4003520 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.578+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 >> v1:172.21.15.204:6789/0 conn(0x7fb9f40c51c0 legacy=0x7fb9f4029800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.578+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 >> v1:172.21.15.22:6789/0 conn(0x7fb9f4034850 legacy=0x7fb9f40c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.578+0000 7fb9f959a640 10 monclient: _finish_hunting 0 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.578+0000 7fb9f959a640 1 monclient: found mon.noname-b 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f4032980 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: _finish_auth 0 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: _check_auth_tickets 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb9f0001690 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: dump: 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9fb59e640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:43:00.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9fb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9f0001860 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 1 -- 172.21.15.22:0/616228650 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb9f0001cb0 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9f959a640 10 monclient: dump: 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.590 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9fb59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.579+0000 7fb9fb59e640 4 set_mon_vals no callback set 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.580+0000 7fb9fb59e640 10 monclient: shutdown 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.580+0000 7fb9fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.580+0000 7fb9fb59e640 1 -- 172.21.15.22:0/616228650 >> v1:172.21.15.22:6790/0 conn(0x7fb9f40c7cd0 legacy=0x7fb9f4029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.580+0000 7fb9fb59e640 1 -- 172.21.15.22:0/616228650 shutdown_connections 2024-03-29T11:43:00.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.580+0000 7fb9fb59e640 1 -- 172.21.15.22:0/616228650 >> 172.21.15.22:0/616228650 conn(0x7fb9f40058f0 msgr2=0x7fb9f4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:00.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.584+0000 7fb9fb59e640 1 -- 172.21.15.22:0/616228650 shutdown_connections 2024-03-29T11:43:00.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.585+0000 7fb9fb59e640 1 -- 172.21.15.22:0/616228650 wait complete. 2024-03-29T11:43:00.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.586+0000 7fb9fb59e640 10 monclient: build_initial_monmap 2024-03-29T11:43:00.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.586+0000 7fb9fb59e640 10 monclient: monmap: 2024-03-29T11:43:00.593 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:00.593 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.586+0000 7fb9fb59e640 1 librados: starting msgr at 2024-03-29T11:43:00.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.586+0000 7fb9fb59e640 1 librados: starting objecter 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.586+0000 7fb9fb59e640 1 Processor -- start 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.587+0000 7fb9fb59e640 1 -- start start 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.587+0000 7fb9fb59e640 1 librados: setting wanted keys 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.587+0000 7fb9fb59e640 1 librados: calling monclient init 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.587+0000 7fb9fb59e640 10 monclient: init 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f40c51c0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f40c7cd0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f40c6a00 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:00.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40c6e40 con 0x7fb9f40c6a00 2024-03-29T11:43:00.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40b2d70 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40b3c10 con 0x7fb9f40c51c0 2024-03-29T11:43:00.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:00.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.588+0000 7fb9fb59e640 10 monclient(hunting): authenticate will time out at 1430.865381s 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.595+0000 7fb9fa59c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb9f40c51c0 0x7fb9f40b1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53875/0 (socket says 172.21.15.22:53875) 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.595+0000 7fb9fa59c640 1 -- 172.21.15.22:0/2022850782 learned_addr learned my addr 172.21.15.22:0/2022850782 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 client.?.objecter ms_handle_connect 0x7fb9f40c51c0 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 client.?.objecter ms_handle_connect 0x7fb9f40c7cd0 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904541085 0 0) 0x7fb9f40b3c10 con 0x7fb9f40c51c0 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:00.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc004eb0 con 0x7fb9f40c51c0 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063353704 0 0) 0x7fb9f40b2d70 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.596+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f40b3c10 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2585844020 0 0) 0x7fb9f40b3c10 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9f40b2d70 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 992722060 0 0) 0x7fb9dc004eb0 con 0x7fb9f40c51c0 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9f40b3c10 con 0x7fb9f40c51c0 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb9ec001490 con 0x7fb9f40c7cd0 2024-03-29T11:43:00.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 10 monclient(hunting): dump: 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:00.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb9fc04fdd0 con 0x7fb9f40c51c0 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.597+0000 7fb9eaffd640 10 monclient(hunting): dump: 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:00.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 10 client.?.objecter ms_handle_connect 0x7fb9f40c6a00 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3352435263 0 0) 0x7fb9f40b3c10 con 0x7fb9f40c51c0 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 >> v1:172.21.15.204:6789/0 conn(0x7fb9f40c7cd0 legacy=0x7fb9f40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 >> v1:172.21.15.22:6789/0 conn(0x7fb9f40c6a00 legacy=0x7fb9f40c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 1 monclient: found mon.c 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f40b3ef0 con 0x7fb9f40c51c0 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:43:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 10 monclient: _renew_subs 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f40b48c0 con 0x7fb9f40c51c0 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 10 monclient: _renew_subs 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f40b2420 con 0x7fb9f40c51c0 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.598+0000 7fb9fb59e640 1 librados: init done 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.600+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9fc04f400 con 0x7fb9f40c51c0 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.600+0000 7fb9eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.600+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fb9fc075710 con 0x7fb9f40c51c0 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.600+0000 7fb9eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.600+0000 7fb9eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.600+0000 7fb9eaffd640 10 monclient: dump: 2024-03-29T11:43:00.611 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:00.612 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:00.612 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.612 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:00.612 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.612 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:00.612 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:00.612 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.600+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fb9fc067bf0 con 0x7fb9f40c51c0 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.600+0000 7fb9ebfff640 4 set_mon_vals no callback set 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.601+0000 7fb9f9d9b640 10 monclient: get_auth_request con 0x7fb9dc06de50 auth_method 0 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.601+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fb9fc0ff490 con 0x7fb9f40c51c0 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.601+0000 7fb9eaffd640 10 client.4568.objecter ms_dispatch 0x7fb9fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.601+0000 7fb9eaffd640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.601+0000 7fb9eaffd640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.601+0000 7fb9eaffd640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:43:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.601+0000 7fb9eaffd640 10 client.4568.objecter ms_handle_connect 0x7fb9dc06de50 2024-03-29T11:43:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.602+0000 7fb9fb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.602+0000 7fb9fb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.602+0000 7fb9fb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:00.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.602+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e0005080 con 0x7fb9f40c51c0 2024-03-29T11:43:00.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.617+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9fc0cadf0 con 0x7fb9f40c51c0 2024-03-29T11:43:00.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.618+0000 7fb9eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.618+0000 7fb9eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:00.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.944+0000 7fb9fb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:00.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.944+0000 7fb9fb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:00.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.944+0000 7fb9fb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:00.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.944+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9e0005330 con 0x7fb9f40c51c0 2024-03-29T11:43:00.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.948+0000 7fb9eaffd640 1 -- 172.21.15.22:0/2022850782 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fb9fc0ccd20 con 0x7fb9f40c51c0 2024-03-29T11:43:00.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.948+0000 7fb9eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:00.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.948+0000 7fb9eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:00.952 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:00.952 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:00.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.952+0000 7fb9fb59e640 10 librados: watch_flush enter 2024-03-29T11:43:00.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.954+0000 7fb9fb59e640 10 librados: watch_flush exit 2024-03-29T11:43:00.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.954+0000 7fb9fb59e640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:43:00.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.954+0000 7fb9fb59e640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:43:00.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.954+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 >> v1:172.21.15.204:6816/3848099522 conn(0x7fb9dc06de50 legacy=0x7fb9dc070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.954+0000 7fb9fb59e640 10 monclient: shutdown 2024-03-29T11:43:00.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.954+0000 7fb9fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:00.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.954+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 >> v1:172.21.15.22:6790/0 conn(0x7fb9f40c51c0 legacy=0x7fb9f40b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.955+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 shutdown_connections 2024-03-29T11:43:00.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.955+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 >> 172.21.15.22:0/2022850782 conn(0x7fb9f40058f0 msgr2=0x7fb9f40043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:00.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.956+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 shutdown_connections 2024-03-29T11:43:00.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.956+0000 7fb9fb59e640 1 -- 172.21.15.22:0/2022850782 wait complete. 2024-03-29T11:43:00.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:00.956+0000 7fb9fb59e640 1 librados: shutdown 2024-03-29T11:43:00.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:07.789 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.938+0000 7fd874e88640 10 monclient: get_monmap_and_config 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.938+0000 7fd874e88640 10 monclient: build_initial_monmap 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.938+0000 7fd874e88640 10 monclient: monmap: 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:07.940224+0000 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:07.940224+0000 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:07.943 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:07.944 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:07.944 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:07.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.939+0000 7fd874e88640 1 Processor -- start 2024-03-29T11:43:07.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.940+0000 7fd874e88640 1 -- start start 2024-03-29T11:43:07.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.940+0000 7fd874e88640 10 monclient: init 2024-03-29T11:43:07.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:07.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:07.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 10 monclient(hunting): picked mon.noname-c con 0x7fd8700b0770 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:07.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 10 monclient(hunting): picked mon.noname-a con 0x7fd87014a700 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:07.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 10 monclient(hunting): picked mon.noname-b con 0x7fd8700ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:07.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8700aba70 con 0x7fd87014a700 2024-03-29T11:43:07.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8700aecb0 con 0x7fd8700b0770 2024-03-29T11:43:07.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8700af150 con 0x7fd8700ac5a0 2024-03-29T11:43:07.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:07.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.941+0000 7fd874e88640 10 monclient(hunting): authenticate will time out at 1438.218327s 2024-03-29T11:43:07.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.948+0000 7fd86dd9b640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fd87014a700 0x7fd8700abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:37843/0 (socket says 172.21.15.22:37843) 2024-03-29T11:43:07.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.948+0000 7fd86dd9b640 1 -- 172.21.15.22:0/2415960144 learned_addr learned my addr 172.21.15.22:0/2415960144 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108099687 0 0) 0x7fd8700aba70 con 0x7fd87014a700 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd858003520 con 0x7fd87014a700 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926226514 0 0) 0x7fd8700aecb0 con 0x7fd8700b0770 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:43:07.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.952+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8700aba70 con 0x7fd8700b0770 2024-03-29T11:43:07.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2512453331 0 0) 0x7fd858003520 con 0x7fd87014a700 2024-03-29T11:43:07.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 >> v1:172.21.15.22:6790/0 conn(0x7fd8700ac5a0 legacy=0x7fd870151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:07.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 >> v1:172.21.15.204:6789/0 conn(0x7fd8700b0770 legacy=0x7fd870149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:07.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 1 monclient: found mon.noname-a 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8700ac9d0 con 0x7fd87014a700 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 10 monclient: _finish_auth 0 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd868001a10 con 0x7fd87014a700 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.953+0000 7fd86d59a640 10 monclient: dump: 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:07.959 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.954+0000 7fd874e88640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.954+0000 7fd874e88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.954+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd868001c30 con 0x7fd87014a700 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.954+0000 7fd86d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.954+0000 7fd86d59a640 1 -- 172.21.15.22:0/2415960144 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd868001fc0 con 0x7fd87014a700 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.954+0000 7fd86d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.954+0000 7fd86d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:07.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.954+0000 7fd86d59a640 10 monclient: dump: 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:07.961 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.955+0000 7fd874e88640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.955+0000 7fd874e88640 4 set_mon_vals no callback set 2024-03-29T11:43:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.956+0000 7fd874e88640 10 monclient: shutdown 2024-03-29T11:43:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.956+0000 7fd874e88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.956+0000 7fd874e88640 1 -- 172.21.15.22:0/2415960144 >> v1:172.21.15.22:6789/0 conn(0x7fd87014a700 legacy=0x7fd8700abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.958+0000 7fd874e88640 1 -- 172.21.15.22:0/2415960144 shutdown_connections 2024-03-29T11:43:07.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.958+0000 7fd874e88640 1 -- 172.21.15.22:0/2415960144 >> 172.21.15.22:0/2415960144 conn(0x7fd87006b660 msgr2=0x7fd87006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:07.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.958+0000 7fd874e88640 1 -- 172.21.15.22:0/2415960144 shutdown_connections 2024-03-29T11:43:07.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.958+0000 7fd874e88640 1 -- 172.21.15.22:0/2415960144 wait complete. 2024-03-29T11:43:07.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.959+0000 7fd874e88640 10 monclient: build_initial_monmap 2024-03-29T11:43:07.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.960+0000 7fd874e88640 10 monclient: monmap: 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.961+0000 7fd874e88640 1 librados: starting msgr at 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.961+0000 7fd874e88640 1 librados: starting objecter 2024-03-29T11:43:07.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.961+0000 7fd874e88640 1 Processor -- start 2024-03-29T11:43:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.962+0000 7fd874e88640 1 -- start start 2024-03-29T11:43:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.962+0000 7fd874e88640 1 librados: setting wanted keys 2024-03-29T11:43:07.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.962+0000 7fd874e88640 1 librados: calling monclient init 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.962+0000 7fd874e88640 10 monclient: init 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.962+0000 7fd874e88640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 10 monclient(hunting): picked mon.noname-b con 0x7fd8700ac5a0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 10 monclient(hunting): picked mon.noname-c con 0x7fd8700b0770 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 10 monclient(hunting): picked mon.noname-a con 0x7fd87014a700 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87011d3c0 con 0x7fd87014a700 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd870147be0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87011ef50 con 0x7fd8700b0770 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:07.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.963+0000 7fd874e88640 10 monclient(hunting): authenticate will time out at 1438.240108s 2024-03-29T11:43:07.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.966+0000 7fd86ed9d640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fd87014a700 0x7fd8701474d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38581/0 (socket says 172.21.15.22:38581) 2024-03-29T11:43:07.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.966+0000 7fd86ed9d640 1 -- 172.21.15.22:0/1612036315 learned_addr learned my addr 172.21.15.22:0/1612036315 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:07.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.966+0000 7fd85effd640 10 client.?.objecter ms_handle_connect 0x7fd87014a700 2024-03-29T11:43:07.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.966+0000 7fd85effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.968+0000 7fd85effd640 10 client.?.objecter ms_handle_connect 0x7fd8700b0770 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.968+0000 7fd85effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.968+0000 7fd85effd640 10 client.?.objecter ms_handle_connect 0x7fd8700ac5a0 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.968+0000 7fd85effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.968+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526334262 0 0) 0x7fd87011ef50 con 0x7fd8700b0770 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:43:07.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd850004eb0 con 0x7fd8700b0770 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633474345 0 0) 0x7fd87011d3c0 con 0x7fd87014a700 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd87011ef50 con 0x7fd87014a700 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952823346 0 0) 0x7fd870147be0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd87011d3c0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 394074059 0 0) 0x7fd850004eb0 con 0x7fd8700b0770 2024-03-29T11:43:07.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd870147be0 con 0x7fd8700b0770 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd868001f40 con 0x7fd8700b0770 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): dump: 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:07.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554424542 0 0) 0x7fd87011d3c0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd850004eb0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd8600017f0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.969+0000 7fd85effd640 10 monclient(hunting): dump: 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:07.977 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3093606510 0 0) 0x7fd850004eb0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 >> v1:172.21.15.22:6790/0 conn(0x7fd8700b0770 legacy=0x7fd870146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 >> v1:172.21.15.22:6789/0 conn(0x7fd87014a700 legacy=0x7fd8701474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 1 monclient: found mon.b 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87011f230 con 0x7fd8700ac5a0 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 10 monclient: _finish_auth 0 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd85effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 10 monclient: _renew_subs 2024-03-29T11:43:07.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:43:07.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87011df80 con 0x7fd8700ac5a0 2024-03-29T11:43:07.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:07.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 10 monclient: _renew_subs 2024-03-29T11:43:07.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:43:07.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8701e7510 con 0x7fd8700ac5a0 2024-03-29T11:43:07.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.970+0000 7fd874e88640 1 librados: init done 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.971+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd860001c80 con 0x7fd8700ac5a0 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.971+0000 7fd85effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.971+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fd8600017f0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.971+0000 7fd85effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.971+0000 7fd85effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.971+0000 7fd85effd640 10 monclient: dump: 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:07.980 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:07.981 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:07.981 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:07.981 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:07.981 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:07.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.972+0000 7fd86cd99640 4 set_mon_vals no callback set 2024-03-29T11:43:07.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.972+0000 7fd874e88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:07.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd874e88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd874e88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fd860019460 con 0x7fd8700ac5a0 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd854005080 con 0x7fd8700ac5a0 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd86dd9b640 10 monclient: get_auth_request con 0x7fd85006e360 auth_method 0 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fd860053000 con 0x7fd8700ac5a0 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd85effd640 10 client.4501.objecter ms_dispatch 0x7fd870000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd85effd640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd85effd640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.973+0000 7fd85effd640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T11:43:07.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.974+0000 7fd85effd640 10 client.4501.objecter ms_handle_connect 0x7fd85006e360 2024-03-29T11:43:07.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.981+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd86004f2a0 con 0x7fd8700ac5a0 2024-03-29T11:43:07.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.981+0000 7fd85effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:07.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:07.981+0000 7fd85effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:08.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.373+0000 7fd874e88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.373+0000 7fd874e88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.373+0000 7fd874e88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:43:08.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.373+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd854005330 con 0x7fd8700ac5a0 2024-03-29T11:43:08.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.374+0000 7fd85effd640 1 -- 172.21.15.22:0/1612036315 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fd86004f440 con 0x7fd8700ac5a0 2024-03-29T11:43:08.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.374+0000 7fd85effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.374+0000 7fd85effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:08.379 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:08.379 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:08.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.379+0000 7fd874e88640 10 librados: watch_flush enter 2024-03-29T11:43:08.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.379+0000 7fd874e88640 10 librados: watch_flush exit 2024-03-29T11:43:08.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.379+0000 7fd874e88640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T11:43:08.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.379+0000 7fd874e88640 10 client.4501.objecter successfully canceled tick 2024-03-29T11:43:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.380+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 >> v1:172.21.15.204:6816/3848099522 conn(0x7fd85006e360 legacy=0x7fd850070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.380+0000 7fd874e88640 10 monclient: shutdown 2024-03-29T11:43:08.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.380+0000 7fd874e88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:08.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.380+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 >> v1:172.21.15.204:6789/0 conn(0x7fd8700ac5a0 legacy=0x7fd87011d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.385+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 shutdown_connections 2024-03-29T11:43:08.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.385+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 >> 172.21.15.22:0/1612036315 conn(0x7fd87006b660 msgr2=0x7fd8700abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.385+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 shutdown_connections 2024-03-29T11:43:08.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.386+0000 7fd874e88640 1 -- 172.21.15.22:0/1612036315 wait complete. 2024-03-29T11:43:08.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:08.386+0000 7fd874e88640 1 librados: shutdown 2024-03-29T11:43:08.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:15.221 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.360+0000 7f3be0045640 10 monclient: get_monmap_and_config 2024-03-29T11:43:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.360+0000 7f3be0045640 10 monclient: build_initial_monmap 2024-03-29T11:43:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.360+0000 7f3be0045640 10 monclient: monmap: 2024-03-29T11:43:15.364 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:15.362197+0000 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:15.362197+0000 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:15.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.362+0000 7f3be0045640 1 Processor -- start 2024-03-29T11:43:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.362+0000 7f3be0045640 1 -- start start 2024-03-29T11:43:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.362+0000 7f3be0045640 10 monclient: init 2024-03-29T11:43:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd80b0770 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd814a700 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd80ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd80aba70 con 0x7f3bd80b0770 2024-03-29T11:43:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd80aecb0 con 0x7f3bd814a700 2024-03-29T11:43:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd80af150 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.363+0000 7f3be0045640 10 monclient(hunting): authenticate will time out at 1445.640936s 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.364+0000 7f3bde5e2640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f3bd80ac5a0 0x7f3bd8151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35612/0 (socket says 172.21.15.22:35612) 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.364+0000 7f3bde5e2640 1 -- 172.21.15.22:0/3286898716 learned_addr learned my addr 172.21.15.22:0/3286898716 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.364+0000 7f3bdcddf640 1 -- 172.21.15.22:0/3286898716 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232415264 0 0) 0x7f3bd80af150 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.364+0000 7f3bdcddf640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.364+0000 7f3bdcddf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.364+0000 7f3bdcddf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.364+0000 7f3bdcddf640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.364+0000 7f3bdcddf640 1 -- 172.21.15.22:0/3286898716 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc4003520 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.365+0000 7f3bdcddf640 1 -- 172.21.15.22:0/3286898716 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236171974 0 0) 0x7f3bc4003520 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.365+0000 7f3bdcddf640 1 -- 172.21.15.22:0/3286898716 >> v1:172.21.15.204:6789/0 conn(0x7f3bd814a700 legacy=0x7f3bd80abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:15.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 1 -- 172.21.15.22:0/3286898716 >> v1:172.21.15.22:6789/0 conn(0x7f3bd80b0770 legacy=0x7f3bd8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:15.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 10 monclient: _finish_hunting 0 2024-03-29T11:43:15.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 1 monclient: found mon.noname-b 2024-03-29T11:43:15.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:15.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:43:15.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 1 -- 172.21.15.22:0/3286898716 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd80ac9d0 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 10 monclient: _finish_auth 0 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 10 monclient: _check_auth_tickets 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3be0045640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3be0045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 1 -- 172.21.15.22:0/3286898716 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3bd4001670 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 10 monclient: dump: 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:15.373 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3be0045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 1 -- 172.21.15.22:0/3286898716 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bd40018d0 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3bdcddf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3be0045640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:15.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.367+0000 7f3be0045640 4 set_mon_vals no callback set 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.368+0000 7f3be0045640 10 monclient: shutdown 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.368+0000 7f3be0045640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.368+0000 7f3be0045640 1 -- 172.21.15.22:0/3286898716 >> v1:172.21.15.22:6790/0 conn(0x7f3bd80ac5a0 legacy=0x7f3bd8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.370+0000 7f3be0045640 1 -- 172.21.15.22:0/3286898716 shutdown_connections 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.370+0000 7f3be0045640 1 -- 172.21.15.22:0/3286898716 >> 172.21.15.22:0/3286898716 conn(0x7f3bd806b660 msgr2=0x7f3bd806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.370+0000 7f3be0045640 1 -- 172.21.15.22:0/3286898716 shutdown_connections 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.370+0000 7f3be0045640 1 -- 172.21.15.22:0/3286898716 wait complete. 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.371+0000 7f3be0045640 10 monclient: build_initial_monmap 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.371+0000 7f3be0045640 10 monclient: monmap: 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:15.375 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:15.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.372+0000 7f3be0045640 1 librados: starting msgr at 2024-03-29T11:43:15.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.372+0000 7f3be0045640 1 librados: starting objecter 2024-03-29T11:43:15.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.372+0000 7f3be0045640 1 Processor -- start 2024-03-29T11:43:15.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.372+0000 7f3be0045640 1 -- start start 2024-03-29T11:43:15.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.372+0000 7f3be0045640 1 librados: setting wanted keys 2024-03-29T11:43:15.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.372+0000 7f3be0045640 1 librados: calling monclient init 2024-03-29T11:43:15.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.372+0000 7f3be0045640 10 monclient: init 2024-03-29T11:43:15.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.373+0000 7f3be0045640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:15.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.373+0000 7f3be0045640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:15.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3be0045640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd80ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:15.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3be0045640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd80b0770 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:15.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3be0045640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd814a700 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:15.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3be0045640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd812a250 con 0x7f3bd814a700 2024-03-29T11:43:15.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3be0045640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd8127d70 con 0x7f3bd80b0770 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3be0045640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd8147a00 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3be0045640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3be0045640 10 monclient(hunting): authenticate will time out at 1445.651488s 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3bde5e2640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f3bd814a700 0x7f3bd81289a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:43680/0 (socket says 172.21.15.22:43680) 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3bde5e2640 1 -- 172.21.15.22:0/78133542 learned_addr learned my addr 172.21.15.22:0/78133542 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3bce7fc640 10 client.?.objecter ms_handle_connect 0x7f3bd814a700 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.374+0000 7f3bce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 client.?.objecter ms_handle_connect 0x7f3bd80ac5a0 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713759931 0 0) 0x7f3bd812a250 con 0x7f3bd814a700 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bbc004eb0 con 0x7f3bd814a700 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177077476 0 0) 0x7f3bd8147a00 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd812a250 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.375+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1446962896 0 0) 0x7f3bbc004eb0 con 0x7f3bd814a700 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bd8147a00 con 0x7f3bd814a700 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3bd4001630 con 0x7f3bd814a700 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:15.383 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905591399 0 0) 0x7f3bd812a250 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bbc004eb0 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3bc8001510 con 0x7f3bd80ac5a0 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3297203417 0 0) 0x7f3bd8147a00 con 0x7f3bd814a700 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 >> v1:172.21.15.22:6790/0 conn(0x7f3bd80ac5a0 legacy=0x7f3bd8146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 >> v1:172.21.15.204:6789/0 conn(0x7f3bd80b0770 legacy=0x7f3bd8128290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 monclient: found mon.a 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd8147b90 con 0x7f3bd814a700 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.376+0000 7f3bce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 10 monclient: _renew_subs 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd8147590 con 0x7f3bd814a700 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:15.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 10 monclient: _renew_subs 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd81df6b0 con 0x7f3bd814a700 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3be0045640 1 librados: init done 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bd4004fd0 con 0x7f3bd814a700 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f3bd40052e0 con 0x7f3bd814a700 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bce7fc640 10 monclient: dump: 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:15.387 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.388 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:15.388 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:15.388 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:15.388 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:15.388 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:15.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.377+0000 7f3bcffff640 4 set_mon_vals no callback set 2024-03-29T11:43:15.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.378+0000 7f3bbbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.378+0000 7f3bbbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.378+0000 7f3bbbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:15.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.378+0000 7f3bbbfff640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb4005080 con 0x7f3bd814a700 2024-03-29T11:43:15.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.380+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f3bd401b2b0 con 0x7f3bd814a700 2024-03-29T11:43:15.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.381+0000 7f3bddde1640 10 monclient: get_auth_request con 0x7f3bbc06dfb0 auth_method 0 2024-03-29T11:43:15.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.381+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f3bd4054d70 con 0x7f3bd814a700 2024-03-29T11:43:15.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.381+0000 7f3bce7fc640 10 client.4542.objecter ms_dispatch 0x7f3bd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:15.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.381+0000 7f3bce7fc640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:15.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.381+0000 7f3bce7fc640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:15.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.381+0000 7f3bce7fc640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:43:15.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.382+0000 7f3bce7fc640 10 client.4542.objecter ms_handle_connect 0x7f3bbc06dfb0 2024-03-29T11:43:15.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.388+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bd4051010 con 0x7f3bd814a700 2024-03-29T11:43:15.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.388+0000 7f3bce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.388+0000 7f3bce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:15.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.748+0000 7f3bbbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.748+0000 7f3bbbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.748+0000 7f3bbbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:15.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.748+0000 7f3bbbfff640 1 -- 172.21.15.22:0/78133542 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bb4005330 con 0x7f3bd814a700 2024-03-29T11:43:15.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.749+0000 7f3bce7fc640 1 -- 172.21.15.22:0/78133542 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f3bd4051010 con 0x7f3bd814a700 2024-03-29T11:43:15.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.749+0000 7f3bce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.749+0000 7f3bce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:15.754 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:15.754 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:15.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 10 librados: watch_flush enter 2024-03-29T11:43:15.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 10 librados: watch_flush exit 2024-03-29T11:43:15.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:43:15.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:43:15.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 1 -- 172.21.15.22:0/78133542 >> v1:172.21.15.204:6816/3848099522 conn(0x7f3bbc06dfb0 legacy=0x7f3bbc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 10 monclient: shutdown 2024-03-29T11:43:15.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:15.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 1 -- 172.21.15.22:0/78133542 >> v1:172.21.15.22:6789/0 conn(0x7f3bd814a700 legacy=0x7f3bd81289a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 1 -- 172.21.15.22:0/78133542 shutdown_connections 2024-03-29T11:43:15.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 1 -- 172.21.15.22:0/78133542 >> 172.21.15.22:0/78133542 conn(0x7f3bd806b660 msgr2=0x7f3bd806d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:15.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 1 -- 172.21.15.22:0/78133542 shutdown_connections 2024-03-29T11:43:15.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 1 -- 172.21.15.22:0/78133542 wait complete. 2024-03-29T11:43:15.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:15.753+0000 7f3bbbfff640 1 librados: shutdown 2024-03-29T11:43:15.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:22.578 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:22.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.722+0000 7f5b29f72640 10 monclient: get_monmap_and_config 2024-03-29T11:43:22.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.722+0000 7f5b29f72640 10 monclient: build_initial_monmap 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.722+0000 7f5b29f72640 10 monclient: monmap: 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:22.724059+0000 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:22.724059+0000 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:22.727 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:22.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.723+0000 7f5b29f72640 1 Processor -- start 2024-03-29T11:43:22.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.724+0000 7f5b29f72640 1 -- start start 2024-03-29T11:43:22.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.724+0000 7f5b29f72640 10 monclient: init 2024-03-29T11:43:22.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.724+0000 7f5b29f72640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:22.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.724+0000 7f5b29f72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.724+0000 7f5b29f72640 10 monclient(hunting): picked mon.noname-c con 0x7f5b241457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b29f72640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2413f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b29f72640 10 monclient(hunting): picked mon.noname-a con 0x7f5b240a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b29f72640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240aaef0 con 0x7f5b240a9d70 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b29f72640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24145bd0 con 0x7f5b241457f0 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b29f72640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24145eb0 con 0x7f5b2413f650 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b29f72640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b29f72640 10 monclient(hunting): authenticate will time out at 1453.002082s 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b22ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5b2413f650 0x7f5b240b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45670/0 (socket says 172.21.15.22:45670) 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b22ffd640 1 -- 172.21.15.22:0/4132116524 learned_addr learned my addr 172.21.15.22:0/4132116524 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831472513 0 0) 0x7f5b24145eb0 con 0x7f5b2413f650 2024-03-29T11:43:22.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b227fc640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.725+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c003520 con 0x7f5b2413f650 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 32265868 0 0) 0x7f5b0c003520 con 0x7f5b2413f650 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 >> v1:172.21.15.204:6789/0 conn(0x7f5b241457f0 legacy=0x7f5b2413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 >> v1:172.21.15.22:6789/0 conn(0x7f5b240a9d70 legacy=0x7f5b2414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 1 monclient: found mon.noname-b 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24146340 con 0x7f5b2413f650 2024-03-29T11:43:22.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b14001690 con 0x7f5b2413f650 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: dump: 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:22.740 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b29f72640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b29f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b14001920 con 0x7f5b2413f650 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 1 -- 172.21.15.22:0/4132116524 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b14001d30 con 0x7f5b2413f650 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b227fc640 10 monclient: dump: 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.741 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b29f72640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.726+0000 7f5b29f72640 4 set_mon_vals no callback set 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.727+0000 7f5b29f72640 10 monclient: shutdown 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.727+0000 7f5b29f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.727+0000 7f5b29f72640 1 -- 172.21.15.22:0/4132116524 >> v1:172.21.15.22:6790/0 conn(0x7f5b2413f650 legacy=0x7f5b240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.727+0000 7f5b29f72640 1 -- 172.21.15.22:0/4132116524 shutdown_connections 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.727+0000 7f5b29f72640 1 -- 172.21.15.22:0/4132116524 >> 172.21.15.22:0/4132116524 conn(0x7f5b24066a00 msgr2=0x7f5b24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:22.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.727+0000 7f5b29f72640 1 -- 172.21.15.22:0/4132116524 shutdown_connections 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.727+0000 7f5b29f72640 1 -- 172.21.15.22:0/4132116524 wait complete. 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.728+0000 7f5b29f72640 10 monclient: build_initial_monmap 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.728+0000 7f5b29f72640 10 monclient: monmap: 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.728+0000 7f5b29f72640 1 librados: starting msgr at 2024-03-29T11:43:22.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.728+0000 7f5b29f72640 1 librados: starting objecter 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.728+0000 7f5b29f72640 1 Processor -- start 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 1 -- start start 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 1 librados: setting wanted keys 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 1 librados: calling monclient init 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 10 monclient: init 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 10 monclient(hunting): picked mon.noname-c con 0x7f5b240a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2413f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 10 monclient(hunting): picked mon.noname-b con 0x7f5b241457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2406fc50 con 0x7f5b2413f650 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24071580 con 0x7f5b241457f0 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240717f0 con 0x7f5b240a9d70 2024-03-29T11:43:22.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.729+0000 7f5b29f72640 10 monclient(hunting): authenticate will time out at 1453.006878s 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b237fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5b240a9d70 0x7f5b241dee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45944/0 (socket says 172.21.15.22:45944) 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b237fe640 1 -- 172.21.15.22:0/1199055496 learned_addr learned my addr 172.21.15.22:0/1199055496 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 client.?.objecter ms_handle_connect 0x7f5b240a9d70 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 client.?.objecter ms_handle_connect 0x7f5b241457f0 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798885084 0 0) 0x7f5b240717f0 con 0x7f5b240a9d70 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b10004eb0 con 0x7f5b240a9d70 2024-03-29T11:43:22.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 client.?.objecter ms_handle_connect 0x7f5b2413f650 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801050771 0 0) 0x7f5b2406fc50 con 0x7f5b2413f650 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240717f0 con 0x7f5b2413f650 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460686627 0 0) 0x7f5b24071580 con 0x7f5b241457f0 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.730+0000 7f5b0bfff640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b2406fc50 con 0x7f5b241457f0 2024-03-29T11:43:22.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107371790 0 0) 0x7f5b10004eb0 con 0x7f5b240a9d70 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b24071580 con 0x7f5b240a9d70 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b180017d0 con 0x7f5b240a9d70 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 10 monclient(hunting): dump: 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:22.747 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3097614449 0 0) 0x7f5b2406fc50 con 0x7f5b241457f0 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b10004eb0 con 0x7f5b241457f0 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b1c001b70 con 0x7f5b241457f0 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 10 monclient(hunting): dump: 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:22.748 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596006663 0 0) 0x7f5b240717f0 con 0x7f5b2413f650 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.731+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b2406fc50 con 0x7f5b2413f650 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650182728 0 0) 0x7f5b24071580 con 0x7f5b240a9d70 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 >> v1:172.21.15.204:6789/0 conn(0x7f5b241457f0 legacy=0x7f5b241e7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 >> v1:172.21.15.22:6789/0 conn(0x7f5b2413f650 legacy=0x7f5b241398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 1 monclient: found mon.c 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2406f890 con 0x7f5b240a9d70 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 10 monclient: _finish_auth 0 2024-03-29T11:43:22.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 10 monclient: _renew_subs 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b24139fb0 con 0x7f5b240a9d70 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 10 monclient: _renew_subs 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b24141c20 con 0x7f5b240a9d70 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b29f72640 1 librados: init done 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b18001ce0 con 0x7f5b240a9d70 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5b18004a00 con 0x7f5b240a9d70 2024-03-29T11:43:22.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b0bfff640 10 monclient: dump: 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.732+0000 7f5b217fa640 4 set_mon_vals no callback set 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.733+0000 7f5b29f72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.734+0000 7f5b29f72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.734+0000 7f5b29f72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:22.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.734+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b04005080 con 0x7f5b240a9d70 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.736+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f5b1801a9d0 con 0x7f5b240a9d70 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.736+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f5b18084920 con 0x7f5b240a9d70 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.736+0000 7f5b0bfff640 10 client.4586.objecter ms_dispatch 0x7f5b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.736+0000 7f5b0bfff640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.736+0000 7f5b0bfff640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.736+0000 7f5b0bfff640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.736+0000 7f5b22ffd640 10 monclient: get_auth_request con 0x7f5b1006e010 auth_method 0 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.737+0000 7f5b0bfff640 10 client.4586.objecter ms_handle_connect 0x7f5b1006e010 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.746+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b18050200 con 0x7f5b240a9d70 2024-03-29T11:43:22.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.746+0000 7f5b0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:22.746+0000 7f5b0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:23.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.109+0000 7f5b29f72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.110+0000 7f5b29f72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.110+0000 7f5b29f72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:23.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.110+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b04005330 con 0x7f5b240a9d70 2024-03-29T11:43:23.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.111+0000 7f5b0bfff640 1 -- 172.21.15.22:0/1199055496 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f5b18052130 con 0x7f5b240a9d70 2024-03-29T11:43:23.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.111+0000 7f5b0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.111+0000 7f5b0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:23.116 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:23.117 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:23.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.114+0000 7f5b29f72640 10 librados: watch_flush enter 2024-03-29T11:43:23.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.114+0000 7f5b29f72640 10 librados: watch_flush exit 2024-03-29T11:43:23.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.114+0000 7f5b29f72640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:43:23.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.114+0000 7f5b29f72640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:43:23.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.114+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 >> v1:172.21.15.204:6816/3848099522 conn(0x7f5b1006e010 legacy=0x7f5b100704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:23.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.114+0000 7f5b29f72640 10 monclient: shutdown 2024-03-29T11:43:23.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.115+0000 7f5b29f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:23.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.115+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 >> v1:172.21.15.22:6790/0 conn(0x7f5b240a9d70 legacy=0x7f5b241dee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:23.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.115+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 shutdown_connections 2024-03-29T11:43:23.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.115+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 >> 172.21.15.22:0/1199055496 conn(0x7f5b24066a00 msgr2=0x7f5b2413dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:23.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.115+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 shutdown_connections 2024-03-29T11:43:23.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.115+0000 7f5b29f72640 1 -- 172.21.15.22:0/1199055496 wait complete. 2024-03-29T11:43:23.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:23.115+0000 7f5b29f72640 1 librados: shutdown 2024-03-29T11:43:23.134 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:29.942 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:30.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.085+0000 7f7652d9d640 10 monclient: get_monmap_and_config 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.085+0000 7f7652d9d640 10 monclient: build_initial_monmap 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.085+0000 7f7652d9d640 10 monclient: monmap: 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:30.086611+0000 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:30.086611+0000 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:30.090 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:30.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.088+0000 7f7652d9d640 1 Processor -- start 2024-03-29T11:43:30.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.088+0000 7f7652d9d640 1 -- start start 2024-03-29T11:43:30.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.088+0000 7f7652d9d640 10 monclient: init 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f764c1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f764c13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f764c0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c0aaef0 con 0x7f764c0a9d70 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c145bd0 con 0x7f764c13f650 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c145eb0 con 0x7f764c1457f0 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:30.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.089+0000 7f7652d9d640 10 monclient(hunting): authenticate will time out at 1460.366576s 2024-03-29T11:43:30.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.092+0000 7f764bfff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f764c13f650 0x7f764c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:36893/0 (socket says 172.21.15.22:36893) 2024-03-29T11:43:30.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.092+0000 7f764bfff640 1 -- 172.21.15.22:0/679075999 learned_addr learned my addr 172.21.15.22:0/679075999 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:30.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133860444 0 0) 0x7f764c145bd0 con 0x7f764c13f650 2024-03-29T11:43:30.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7634003520 con 0x7f764c13f650 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238799690 0 0) 0x7f764c0aaef0 con 0x7f764c0a9d70 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c145bd0 con 0x7f764c0a9d70 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3414875728 0 0) 0x7f764c145bd0 con 0x7f764c0a9d70 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 >> v1:172.21.15.22:6790/0 conn(0x7f764c1457f0 legacy=0x7f764c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 >> v1:172.21.15.204:6789/0 conn(0x7f764c13f650 legacy=0x7f764c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:43:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.096+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c146340 con 0x7f764c0a9d70 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f76440019b0 con 0x7f764c0a9d70 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient: dump: 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:30.116 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f7652d9d640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f7652d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f765133a640 1 -- 172.21.15.22:0/679075999 >> v1:172.21.15.22:6789/0 conn(0x7f764c0a9d70 legacy=0x7f764c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f765133a640 1 -- 172.21.15.22:0/679075999 >> v1:172.21.15.22:6789/0 conn(0x7f764c0a9d70 legacy=0x7f764c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f765133a640 1 --1- 172.21.15.22:0/679075999 >> v1:172.21.15.22:6789/0 conn(0x7f764c0a9d70 0x7f764c14d8c0 :-1 s=OPENED pgs=896400 cs=1 l=1).handle_message read tag failed 2024-03-29T11:43:30.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f765133a640 1 --1- 172.21.15.22:0/679075999 >> v1:172.21.15.22:6789/0 conn(0x7f764c0a9d70 0x7f764c14d8c0 :-1 s=OPENED pgs=896400 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.22:6789/0 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 >> v1:172.21.15.22:6789/0 conn(0x7f764c0a9d70 legacy=0x7f764c14d8c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient(hunting): picked mon.a con 0x7f7634004ce0 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient(hunting): picked mon.c con 0x7f764c1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient(hunting): picked mon.b con 0x7f764c13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f764c145bd0 con 0x7f7634004ce0 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7634004460 con 0x7f764c13f650 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f76340110e0 con 0x7f764c1457f0 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:30.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.22:6789/0 2024-03-29T11:43:30.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263094631 0 0) 0x7f76340110e0 con 0x7f764c1457f0 2024-03-29T11:43:30.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:43:30.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 --> v1:172.21.15.22:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f7634011270 con 0x7f764c1457f0 2024-03-29T11:43:30.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 974575009 0 0) 0x7f7634011270 con 0x7f764c1457f0 2024-03-29T11:43:30.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.097+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 >> v1:172.21.15.204:6789/0 conn(0x7f764c13f650 legacy=0x7f763400d9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.100+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 >> v1:172.21.15.22:6789/0 conn(0x7f7634004ce0 legacy=0x7f76340050c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 1 monclient: found mon.c 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f7634012340 con 0x7f764c1457f0 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:30.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7640001890 con 0x7f764c1457f0 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 10 monclient: dump: 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:30.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 1 -- 172.21.15.22:0/679075999 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76400019f0 con 0x7f764c1457f0 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f764b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f7652d9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.101+0000 7f7652d9d640 4 set_mon_vals no callback set 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.102+0000 7f7652d9d640 10 monclient: shutdown 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.102+0000 7f7652d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.102+0000 7f7652d9d640 1 -- 172.21.15.22:0/679075999 >> v1:172.21.15.22:6790/0 conn(0x7f764c1457f0 legacy=0x7f76340038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.103+0000 7f7652d9d640 1 -- 172.21.15.22:0/679075999 shutdown_connections 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.103+0000 7f7652d9d640 1 -- 172.21.15.22:0/679075999 >> 172.21.15.22:0/679075999 conn(0x7f764c066a00 msgr2=0x7f764c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.105+0000 7f7652d9d640 1 -- 172.21.15.22:0/679075999 shutdown_connections 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.107+0000 7f7652d9d640 1 -- 172.21.15.22:0/679075999 wait complete. 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.108+0000 7f7652d9d640 10 monclient: build_initial_monmap 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.108+0000 7f7652d9d640 10 monclient: monmap: 2024-03-29T11:43:30.122 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.108+0000 7f7652d9d640 1 librados: starting msgr at 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.108+0000 7f7652d9d640 1 librados: starting objecter 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.108+0000 7f7652d9d640 1 Processor -- start 2024-03-29T11:43:30.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.109+0000 7f7652d9d640 1 -- start start 2024-03-29T11:43:30.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.109+0000 7f7652d9d640 1 librados: setting wanted keys 2024-03-29T11:43:30.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.109+0000 7f7652d9d640 1 librados: calling monclient init 2024-03-29T11:43:30.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.109+0000 7f7652d9d640 10 monclient: init 2024-03-29T11:43:30.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.111+0000 7f7652d9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:30.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.111+0000 7f7652d9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:30.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.111+0000 7f7652d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7634004ce0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:30.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.112+0000 7f7652d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f764c13f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:30.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.113+0000 7f7652d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f764c1457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:30.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.113+0000 7f7652d9d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c144fc0 con 0x7f764c13f650 2024-03-29T11:43:30.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.113+0000 7f7652d9d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c135770 con 0x7f764c1457f0 2024-03-29T11:43:30.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.113+0000 7f7652d9d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c13e0a0 con 0x7f7634004ce0 2024-03-29T11:43:30.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.113+0000 7f7652d9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:30.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.113+0000 7f7652d9d640 10 monclient(hunting): authenticate will time out at 1460.391004s 2024-03-29T11:43:30.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.115+0000 7f7650b39640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f7634004ce0 0x7f764c1d6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:38972/0 (socket says 172.21.15.22:38972) 2024-03-29T11:43:30.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.115+0000 7f7650b39640 1 -- 172.21.15.22:0/1042983702 learned_addr learned my addr 172.21.15.22:0/1042983702 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:30.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 client.?.objecter ms_handle_connect 0x7f7634004ce0 2024-03-29T11:43:30.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 client.?.objecter ms_handle_connect 0x7f764c13f650 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524735246 0 0) 0x7f764c13e0a0 con 0x7f7634004ce0 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7638004eb0 con 0x7f7634004ce0 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35974905 0 0) 0x7f764c144fc0 con 0x7f764c13f650 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:43:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.118+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c13e0a0 con 0x7f764c13f650 2024-03-29T11:43:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2508826752 0 0) 0x7f764c13e0a0 con 0x7f764c13f650 2024-03-29T11:43:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f764c144fc0 con 0x7f764c13f650 2024-03-29T11:43:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2763456505 0 0) 0x7f7638004eb0 con 0x7f7634004ce0 2024-03-29T11:43:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f764c13e0a0 con 0x7f7634004ce0 2024-03-29T11:43:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7640001850 con 0x7f764c13f650 2024-03-29T11:43:30.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:30.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient(hunting): dump: 2024-03-29T11:43:30.135 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:30.135 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:30.135 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.135 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.135 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.135 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:30.135 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f763c001950 con 0x7f7634004ce0 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient(hunting): dump: 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:30.136 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3035421180 0 0) 0x7f764c144fc0 con 0x7f764c13f650 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 >> v1:172.21.15.22:6790/0 conn(0x7f7634004ce0 legacy=0x7f764c1d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 >> v1:172.21.15.204:6789/0 conn(0x7f764c1457f0 legacy=0x7f764c135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 monclient: found mon.a 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c13e230 con 0x7f764c13f650 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.119+0000 7f76497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.120+0000 7f7652d9d640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.120+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76400047e0 con 0x7f764c13f650 2024-03-29T11:43:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.120+0000 7f7652d9d640 10 monclient: _renew_subs 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.120+0000 7f7652d9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.120+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764c144ac0 con 0x7f764c13f650 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.120+0000 7f7652d9d640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f76497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f7640004af0 con 0x7f764c13f650 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f76497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f76497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f76497fa640 10 monclient: dump: 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:30.139 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f764affd640 4 set_mon_vals no callback set 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f7652d9d640 10 monclient: _renew_subs 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f7652d9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f764c140840 con 0x7f764c13f650 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.121+0000 7f7652d9d640 1 librados: init done 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.122+0000 7f7652d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.123+0000 7f7652d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.123+0000 7f7652d9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:30.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.123+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764c075f10 con 0x7f764c13f650 2024-03-29T11:43:30.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.125+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f764001aac0 con 0x7f764c13f650 2024-03-29T11:43:30.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.127+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f7640084890 con 0x7f764c13f650 2024-03-29T11:43:30.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.127+0000 7f76497fa640 10 client.4554.objecter ms_dispatch 0x7f764c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:30.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.127+0000 7f76497fa640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:30.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.127+0000 7f76497fa640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:30.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.127+0000 7f76497fa640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:43:30.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.132+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7640050170 con 0x7f764c13f650 2024-03-29T11:43:30.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.132+0000 7f76497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.132+0000 7f76497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:30.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.132+0000 7f7650b39640 10 monclient: get_auth_request con 0x7f763806dfe0 auth_method 0 2024-03-29T11:43:30.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.132+0000 7f76497fa640 10 client.4554.objecter ms_handle_connect 0x7f763806dfe0 2024-03-29T11:43:30.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.472+0000 7f7652d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.472+0000 7f7652d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.472+0000 7f7652d9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:30.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.472+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f764c0ab170 con 0x7f764c13f650 2024-03-29T11:43:30.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.473+0000 7f76497fa640 1 -- 172.21.15.22:0/1042983702 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f76400520a0 con 0x7f764c13f650 2024-03-29T11:43:30.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.474+0000 7f76497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.474+0000 7f76497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:30.478 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:30.478 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:30.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.478+0000 7f7652d9d640 10 librados: watch_flush enter 2024-03-29T11:43:30.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.479+0000 7f7652d9d640 10 librados: watch_flush exit 2024-03-29T11:43:30.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.479+0000 7f7652d9d640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:43:30.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.479+0000 7f7652d9d640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:43:30.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 >> v1:172.21.15.204:6816/3848099522 conn(0x7f763806dfe0 legacy=0x7f76380704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 10 monclient: shutdown 2024-03-29T11:43:30.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:30.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 >> v1:172.21.15.22:6789/0 conn(0x7f764c13f650 legacy=0x7f764c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 shutdown_connections 2024-03-29T11:43:30.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 >> 172.21.15.22:0/1042983702 conn(0x7f764c066a00 msgr2=0x7f764c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:30.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 shutdown_connections 2024-03-29T11:43:30.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 1 -- 172.21.15.22:0/1042983702 wait complete. 2024-03-29T11:43:30.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:30.480+0000 7f7652d9d640 1 librados: shutdown 2024-03-29T11:43:30.502 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:37.309 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.456+0000 7f5241f3a640 10 monclient: get_monmap_and_config 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.456+0000 7f5241f3a640 10 monclient: build_initial_monmap 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.456+0000 7f5241f3a640 10 monclient: monmap: 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:37.457021+0000 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:37.457021+0000 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:37.460 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:37.461 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:37.461 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:37.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.457+0000 7f5241f3a640 1 Processor -- start 2024-03-29T11:43:37.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.458+0000 7f5241f3a640 1 -- start start 2024-03-29T11:43:37.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.458+0000 7f5241f3a640 10 monclient: init 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f523c0b0860 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f523c13ef30 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f523c0af8e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f523bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f523c0af8e0 0x7f523c149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45668/0 (socket says 172.21.15.22:45668) 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f523bfff640 1 -- 172.21.15.22:0/917614823 learned_addr learned my addr 172.21.15.22:0/917614823 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 1 -- 172.21.15.22:0/917614823 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0b0c40 con 0x7f523c0b0860 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 1 -- 172.21.15.22:0/917614823 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0aed10 con 0x7f523c13ef30 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 1 -- 172.21.15.22:0/917614823 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0afd50 con 0x7f523c0af8e0 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:37.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.459+0000 7f5241f3a640 10 monclient(hunting): authenticate will time out at 1467.735956s 2024-03-29T11:43:37.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.461+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008389567 0 0) 0x7f523c0afd50 con 0x7f523c0af8e0 2024-03-29T11:43:37.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.461+0000 7f523a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.461+0000 7f523a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.461+0000 7f523a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.461+0000 7f523a7fc640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:43:37.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.461+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5224003520 con 0x7f523c0af8e0 2024-03-29T11:43:37.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.462+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145555349 0 0) 0x7f5224003520 con 0x7f523c0af8e0 2024-03-29T11:43:37.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.462+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 >> v1:172.21.15.204:6789/0 conn(0x7f523c13ef30 legacy=0x7f523c0af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 >> v1:172.21.15.22:6789/0 conn(0x7f523c0b0860 legacy=0x7f523c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0ab1b0 con 0x7f523c0af8e0 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5234001a10 con 0x7f523c0af8e0 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f5241f3a640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f5241f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:37.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 10 monclient: dump: 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f5241f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5234001be0 con 0x7f523c0af8e0 2024-03-29T11:43:37.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.464+0000 7f523a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.465+0000 7f523a7fc640 1 -- 172.21.15.22:0/917614823 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5234002030 con 0x7f523c0af8e0 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.465+0000 7f5241f3a640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.465+0000 7f5241f3a640 4 set_mon_vals no callback set 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.465+0000 7f523a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.465+0000 7f523a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.465+0000 7f523a7fc640 10 monclient: dump: 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:37.471 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:37.472 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:37.472 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:37.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.466+0000 7f5241f3a640 10 monclient: shutdown 2024-03-29T11:43:37.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.466+0000 7f5241f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:37.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.466+0000 7f5241f3a640 1 -- 172.21.15.22:0/917614823 >> v1:172.21.15.22:6790/0 conn(0x7f523c0af8e0 legacy=0x7f523c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.467+0000 7f5241f3a640 1 -- 172.21.15.22:0/917614823 shutdown_connections 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.467+0000 7f5241f3a640 1 -- 172.21.15.22:0/917614823 >> 172.21.15.22:0/917614823 conn(0x7f523c06b660 msgr2=0x7f523c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.468+0000 7f5241f3a640 1 -- 172.21.15.22:0/917614823 shutdown_connections 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.468+0000 7f5241f3a640 1 -- 172.21.15.22:0/917614823 wait complete. 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.470+0000 7f5241f3a640 10 monclient: build_initial_monmap 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.470+0000 7f5241f3a640 10 monclient: monmap: 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:37.473 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:37.474 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:37.474 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:37.474 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:37.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.471+0000 7f5241f3a640 1 librados: starting msgr at 2024-03-29T11:43:37.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.471+0000 7f5241f3a640 1 librados: starting objecter 2024-03-29T11:43:37.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.471+0000 7f5241f3a640 1 Processor -- start 2024-03-29T11:43:37.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.472+0000 7f5241f3a640 1 -- start start 2024-03-29T11:43:37.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.472+0000 7f5241f3a640 1 librados: setting wanted keys 2024-03-29T11:43:37.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.472+0000 7f5241f3a640 1 librados: calling monclient init 2024-03-29T11:43:37.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.472+0000 7f5241f3a640 10 monclient: init 2024-03-29T11:43:37.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.473+0000 7f5241f3a640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:37.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.473+0000 7f5241f3a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:37.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.474+0000 7f523b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f523c0af8e0 0x7f523c1d2ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47044/0 (socket says 172.21.15.22:47044) 2024-03-29T11:43:37.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.474+0000 7f523b7fe640 1 -- 172.21.15.22:0/345710934 learned_addr learned my addr 172.21.15.22:0/345710934 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:37.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.474+0000 7f5241f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f523c0af8e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.474+0000 7f5223fff640 10 client.?.objecter ms_handle_connect 0x7f523c0af8e0 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.475+0000 7f5223fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.475+0000 7f5241f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f523c0b0860 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.475+0000 7f5241f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f523c13ef30 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.475+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0b3bb0 con 0x7f523c13ef30 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.475+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c1d3860 con 0x7f523c0b0860 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.475+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c1d3d80 con 0x7f523c0af8e0 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.475+0000 7f5241f3a640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:37.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.475+0000 7f5241f3a640 10 monclient(hunting): authenticate will time out at 1467.751688s 2024-03-29T11:43:37.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.476+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232769044 0 0) 0x7f523c1d3d80 con 0x7f523c0af8e0 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.476+0000 7f5223fff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.476+0000 7f5223fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.476+0000 7f5223fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.476+0000 7f5223fff640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.476+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5228004eb0 con 0x7f523c0af8e0 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.477+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3021921795 0 0) 0x7f5228004eb0 con 0x7f523c0af8e0 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.477+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f523c1d3d80 con 0x7f523c0af8e0 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.477+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f522c001930 con 0x7f523c0af8e0 2024-03-29T11:43:37.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.477+0000 7f5223fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.477+0000 7f5223fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.477+0000 7f5223fff640 10 monclient(hunting): dump: 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:37.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.477+0000 7f5223fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:37.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.480+0000 7f5223fff640 10 client.?.objecter ms_handle_connect 0x7f523c0b0860 2024-03-29T11:43:37.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.480+0000 7f5223fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:37.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.480+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2316698502 0 0) 0x7f523c1d3d80 con 0x7f523c0af8e0 2024-03-29T11:43:37.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.480+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 >> v1:172.21.15.204:6789/0 conn(0x7f523c0b0860 legacy=0x7f523c0b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.480+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 >> v1:172.21.15.22:6789/0 conn(0x7f523c13ef30 legacy=0x7f523c1de9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 1 monclient: found mon.c 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c06d9d0 con 0x7f523c0af8e0 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 10 monclient: _finish_auth 0 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 10 client.?.objecter ms_handle_connect 0x7f523c13ef30 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5241f3a640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5241f3a640 10 monclient: _renew_subs 2024-03-29T11:43:37.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5223fff640 10 client.4598.objecter resend_mon_ops 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5241f3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.481+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c06dc20 con 0x7f523c0af8e0 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.482+0000 7f5241f3a640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.482+0000 7f5241f3a640 10 monclient: _renew_subs 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.482+0000 7f5241f3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.482+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c1387b0 con 0x7f523c0af8e0 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.482+0000 7f5241f3a640 1 librados: init done 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.482+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f522c001cb0 con 0x7f523c0af8e0 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.482+0000 7f5223fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.482+0000 7f5239ffb640 4 set_mon_vals no callback set 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.483+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f522c003a60 con 0x7f523c0af8e0 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.483+0000 7f5223fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.483+0000 7f5223fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:37.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.483+0000 7f5223fff640 10 monclient: dump: 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:37.488 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:37.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.484+0000 7f5241f3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.484+0000 7f5241f3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.484+0000 7f5241f3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:37.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.484+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c005080 con 0x7f523c0af8e0 2024-03-29T11:43:37.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.485+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f522c003bc0 con 0x7f523c0af8e0 2024-03-29T11:43:37.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.486+0000 7f523affd640 10 monclient: get_auth_request con 0x7f522806dbd0 auth_method 0 2024-03-29T11:43:37.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.486+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f522c084b80 con 0x7f523c0af8e0 2024-03-29T11:43:37.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.487+0000 7f5223fff640 10 client.4598.objecter ms_dispatch 0x7f523c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:37.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.487+0000 7f5223fff640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:37.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.487+0000 7f5223fff640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:37.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.487+0000 7f5223fff640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:43:37.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.487+0000 7f5223fff640 10 client.4598.objecter ms_handle_connect 0x7f522806dbd0 2024-03-29T11:43:37.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.496+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f522c050460 con 0x7f523c0af8e0 2024-03-29T11:43:37.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.496+0000 7f5223fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.496+0000 7f5223fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.830+0000 7f5241f3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.830+0000 7f5241f3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.830+0000 7f5241f3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.830+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f521c005330 con 0x7f523c0af8e0 2024-03-29T11:43:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.831+0000 7f5223fff640 1 -- 172.21.15.22:0/345710934 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f522c052390 con 0x7f523c0af8e0 2024-03-29T11:43:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.831+0000 7f5223fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.831+0000 7f5223fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:37.837 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:37.837 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 10 librados: watch_flush enter 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 10 librados: watch_flush exit 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 >> v1:172.21.15.204:6816/3848099522 conn(0x7f522806dbd0 legacy=0x7f5228070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 10 monclient: shutdown 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 >> v1:172.21.15.22:6790/0 conn(0x7f523c0af8e0 legacy=0x7f523c1d2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 shutdown_connections 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.834+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 >> 172.21.15.22:0/345710934 conn(0x7f523c06b660 msgr2=0x7f523c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.835+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 shutdown_connections 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.835+0000 7f5241f3a640 1 -- 172.21.15.22:0/345710934 wait complete. 2024-03-29T11:43:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:37.835+0000 7f5241f3a640 1 librados: shutdown 2024-03-29T11:43:37.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:44.660 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:44.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.810+0000 7fc2d9afb640 10 monclient: get_monmap_and_config 2024-03-29T11:43:44.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.810+0000 7fc2d9afb640 10 monclient: build_initial_monmap 2024-03-29T11:43:44.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.810+0000 7fc2d9afb640 10 monclient: monmap: 2024-03-29T11:43:44.819 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:44.810698+0000 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:44.810698+0000 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:44.820 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.812+0000 7fc2d9afb640 1 Processor -- start 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.812+0000 7fc2d9afb640 1 -- start start 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.812+0000 7fc2d9afb640 10 monclient: init 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 10 monclient(hunting): picked mon.noname-a con 0x7fc2d40a9ac0 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 10 monclient(hunting): picked mon.noname-b con 0x7fc2d413ee90 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 10 monclient(hunting): picked mon.noname-c con 0x7fc2d40b4100 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d40af6f0 con 0x7fc2d40a9ac0 2024-03-29T11:43:44.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d40aac80 con 0x7fc2d40b4100 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d40aaf60 con 0x7fc2d413ee90 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.813+0000 7fc2d9afb640 10 monclient(hunting): authenticate will time out at 1475.089812s 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.815+0000 7fc2d2ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc2d413ee90 0x7fc2d413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49620/0 (socket says 172.21.15.22:49620) 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.815+0000 7fc2d2ffd640 1 -- 172.21.15.22:0/1211813355 learned_addr learned my addr 172.21.15.22:0/1211813355 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961685580 0 0) 0x7fc2d40aac80 con 0x7fc2d40b4100 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2bc003520 con 0x7fc2d40b4100 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2665916768 0 0) 0x7fc2bc003520 con 0x7fc2d40b4100 2024-03-29T11:43:44.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 >> v1:172.21.15.22:6790/0 conn(0x7fc2d413ee90 legacy=0x7fc2d413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 >> v1:172.21.15.22:6789/0 conn(0x7fc2d40a9ac0 legacy=0x7fc2d40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 1 monclient: found mon.noname-c 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d413e970 con 0x7fc2d40b4100 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d9afb640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d9afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc2c8001690 con 0x7fc2d40b4100 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d27fc640 10 monclient: dump: 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.818+0000 7fc2d9afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2c80018b0 con 0x7fc2d40b4100 2024-03-29T11:43:44.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d9afb640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d9afb640 4 set_mon_vals no callback set 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d27fc640 1 -- 172.21.15.22:0/1211813355 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc2c8001c40 con 0x7fc2d40b4100 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d27fc640 10 monclient: dump: 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:44.831 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d9afb640 10 monclient: shutdown 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d9afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.819+0000 7fc2d9afb640 1 -- 172.21.15.22:0/1211813355 >> v1:172.21.15.204:6789/0 conn(0x7fc2d40b4100 legacy=0x7fc2d414a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.820+0000 7fc2d9afb640 1 -- 172.21.15.22:0/1211813355 shutdown_connections 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.820+0000 7fc2d9afb640 1 -- 172.21.15.22:0/1211813355 >> 172.21.15.22:0/1211813355 conn(0x7fc2d40b35e0 msgr2=0x7fc2d40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.820+0000 7fc2d9afb640 1 -- 172.21.15.22:0/1211813355 shutdown_connections 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.820+0000 7fc2d9afb640 1 -- 172.21.15.22:0/1211813355 wait complete. 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.821+0000 7fc2d9afb640 10 monclient: build_initial_monmap 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.821+0000 7fc2d9afb640 10 monclient: monmap: 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:44.832 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.821+0000 7fc2d9afb640 1 librados: starting msgr at 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.821+0000 7fc2d9afb640 1 librados: starting objecter 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.821+0000 7fc2d9afb640 1 Processor -- start 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.822+0000 7fc2d9afb640 1 -- start start 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.822+0000 7fc2d9afb640 1 librados: setting wanted keys 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.822+0000 7fc2d9afb640 1 librados: calling monclient init 2024-03-29T11:43:44.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.822+0000 7fc2d9afb640 10 monclient: init 2024-03-29T11:43:44.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.823+0000 7fc2d9afb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:44.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.823+0000 7fc2d9afb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:44.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.823+0000 7fc2d9afb640 10 monclient(hunting): picked mon.noname-b con 0x7fc2d40a9ac0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:44.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.823+0000 7fc2d9afb640 10 monclient(hunting): picked mon.noname-c con 0x7fc2d40b4100 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:44.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.823+0000 7fc2d2ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc2d40b4100 0x7fc2d41db9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50122/0 (socket says 172.21.15.22:50122) 2024-03-29T11:43:44.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.823+0000 7fc2d2ffd640 1 -- 172.21.15.22:0/2220086933 learned_addr learned my addr 172.21.15.22:0/2220086933 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:44.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.823+0000 7fc2bbfff640 10 client.?.objecter ms_handle_connect 0x7fc2d40b4100 2024-03-29T11:43:44.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.823+0000 7fc2bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.826+0000 7fc2d9afb640 10 monclient(hunting): picked mon.noname-a con 0x7fc2d413ee90 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.826+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d41df840 con 0x7fc2d413ee90 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.826+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d41dc810 con 0x7fc2d40a9ac0 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.826+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d4140640 con 0x7fc2d40b4100 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.826+0000 7fc2d9afb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.826+0000 7fc2d9afb640 10 monclient(hunting): authenticate will time out at 1475.102579s 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163267153 0 0) 0x7fc2d4140640 con 0x7fc2d40b4100 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c0004eb0 con 0x7fc2d40b4100 2024-03-29T11:43:44.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 10 client.?.objecter ms_handle_connect 0x7fc2d413ee90 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 10 client.?.objecter ms_handle_connect 0x7fc2d40a9ac0 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.827+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2431172213 0 0) 0x7fc2c0004eb0 con 0x7fc2d40b4100 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2d4140640 con 0x7fc2d40b4100 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653750655 0 0) 0x7fc2d41df840 con 0x7fc2d413ee90 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c0004eb0 con 0x7fc2d413ee90 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc2c4001890 con 0x7fc2d40b4100 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): dump: 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634049380 0 0) 0x7fc2d41dc810 con 0x7fc2d40a9ac0 2024-03-29T11:43:44.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d41df840 con 0x7fc2d40a9ac0 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064818354 0 0) 0x7fc2c0004eb0 con 0x7fc2d413ee90 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2c0005b20 con 0x7fc2d413ee90 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc2c8001a20 con 0x7fc2d413ee90 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): dump: 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:44.840 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 792213822 0 0) 0x7fc2d41df840 con 0x7fc2d40a9ac0 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2c0004eb0 con 0x7fc2d40a9ac0 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc2cc001b60 con 0x7fc2d40a9ac0 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.828+0000 7fc2bbfff640 10 monclient(hunting): dump: 2024-03-29T11:43:44.841 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1743600927 0 0) 0x7fc2c0005b20 con 0x7fc2d413ee90 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 >> v1:172.21.15.22:6790/0 conn(0x7fc2d40b4100 legacy=0x7fc2d41db9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 >> v1:172.21.15.204:6789/0 conn(0x7fc2d40a9ac0 legacy=0x7fc2d4139550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:44.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 1 monclient: found mon.a 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d413f650 con 0x7fc2d413ee90 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 10 monclient: _renew_subs 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d413f8a0 con 0x7fc2d413ee90 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 10 monclient: _renew_subs 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:44.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d413e0f0 con 0x7fc2d413ee90 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.829+0000 7fc2d9afb640 1 librados: init done 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.831+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2c8001c80 con 0x7fc2d413ee90 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.831+0000 7fc2bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.831+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fc2c8003b00 con 0x7fc2d413ee90 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.831+0000 7fc2bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.831+0000 7fc2bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.831+0000 7fc2bbfff640 10 monclient: dump: 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:44.844 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.831+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fc2c8019ad0 con 0x7fc2d413ee90 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.832+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fc2c80838e0 con 0x7fc2d413ee90 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.832+0000 7fc2bbfff640 10 client.4563.objecter ms_dispatch 0x7fc2d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.832+0000 7fc2bbfff640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.832+0000 7fc2bbfff640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.832+0000 7fc2bbfff640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.832+0000 7fc2d17fa640 4 set_mon_vals no callback set 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.833+0000 7fc2d9afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.833+0000 7fc2d9afb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.833+0000 7fc2d9afb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:44.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.833+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b4005080 con 0x7fc2d413ee90 2024-03-29T11:43:44.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.833+0000 7fc2d37fe640 10 monclient: get_auth_request con 0x7fc2c006e5f0 auth_method 0 2024-03-29T11:43:44.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.834+0000 7fc2bbfff640 10 client.4563.objecter ms_handle_connect 0x7fc2c006e5f0 2024-03-29T11:43:44.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.841+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2c804f240 con 0x7fc2d413ee90 2024-03-29T11:43:44.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.841+0000 7fc2bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:44.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:44.841+0000 7fc2bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:45.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.180+0000 7fc2d9afb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.180+0000 7fc2d9afb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.180+0000 7fc2d9afb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:45.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.180+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2b4005330 con 0x7fc2d413ee90 2024-03-29T11:43:45.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.181+0000 7fc2bbfff640 1 -- 172.21.15.22:0/2220086933 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fc2c8051170 con 0x7fc2d413ee90 2024-03-29T11:43:45.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.181+0000 7fc2bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.181+0000 7fc2bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:45.186 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:45.186 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:45.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.186+0000 7fc2d9afb640 10 librados: watch_flush enter 2024-03-29T11:43:45.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.187+0000 7fc2d9afb640 10 librados: watch_flush exit 2024-03-29T11:43:45.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.187+0000 7fc2d9afb640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T11:43:45.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.187+0000 7fc2d9afb640 10 client.4563.objecter successfully canceled tick 2024-03-29T11:43:45.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.187+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 >> v1:172.21.15.204:6816/3848099522 conn(0x7fc2c006e5f0 legacy=0x7fc2c0070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:45.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.187+0000 7fc2d9afb640 10 monclient: shutdown 2024-03-29T11:43:45.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.187+0000 7fc2d9afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:45.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.187+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 >> v1:172.21.15.22:6789/0 conn(0x7fc2d413ee90 legacy=0x7fc2d41dc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:45.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.189+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 shutdown_connections 2024-03-29T11:43:45.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.189+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 >> 172.21.15.22:0/2220086933 conn(0x7fc2d40b35e0 msgr2=0x7fc2d40b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:45.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.189+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 shutdown_connections 2024-03-29T11:43:45.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.190+0000 7fc2d9afb640 1 -- 172.21.15.22:0/2220086933 wait complete. 2024-03-29T11:43:45.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:45.190+0000 7fc2d9afb640 1 librados: shutdown 2024-03-29T11:43:45.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:52.018 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.165+0000 7fe33da99640 10 monclient: get_monmap_and_config 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.165+0000 7fe33da99640 10 monclient: build_initial_monmap 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.165+0000 7fe33da99640 10 monclient: monmap: 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:52.166881+0000 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:52.166881+0000 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:52.170 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:52.171 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:52.171 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:52.171 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:52.171 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:52.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.166+0000 7fe33da99640 1 Processor -- start 2024-03-29T11:43:52.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.167+0000 7fe33da99640 1 -- start start 2024-03-29T11:43:52.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.167+0000 7fe33da99640 10 monclient: init 2024-03-29T11:43:52.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:52.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:52.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 10 monclient(hunting): picked mon.noname-b con 0x7fe3380aad20 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:52.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 10 monclient(hunting): picked mon.noname-c con 0x7fe33806d8c0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:52.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 10 monclient(hunting): picked mon.noname-a con 0x7fe3380afc40 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:52.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3380af4d0 con 0x7fe3380afc40 2024-03-29T11:43:52.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3380af7b0 con 0x7fe33806d8c0 2024-03-29T11:43:52.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3380acab0 con 0x7fe3380aad20 2024-03-29T11:43:52.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:52.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.168+0000 7fe33da99640 10 monclient(hunting): authenticate will time out at 1482.445169s 2024-03-29T11:43:52.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.169+0000 7fe33ca97640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe3380aad20 0x7fe3380ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41866/0 (socket says 172.21.15.22:41866) 2024-03-29T11:43:52.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.169+0000 7fe33ca97640 1 -- 172.21.15.22:0/448995537 learned_addr learned my addr 172.21.15.22:0/448995537 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:52.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.171+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398868362 0 0) 0x7fe3380acab0 con 0x7fe3380aad20 2024-03-29T11:43:52.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.171+0000 7fe3377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.171+0000 7fe3377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.171+0000 7fe3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.171+0000 7fe3377fe640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:43:52.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.171+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe328003520 con 0x7fe3380aad20 2024-03-29T11:43:52.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.173+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202831715 0 0) 0x7fe3380af7b0 con 0x7fe33806d8c0 2024-03-29T11:43:52.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.173+0000 7fe3377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.173+0000 7fe3377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.173+0000 7fe3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.173+0000 7fe3377fe640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:43:52.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.173+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe328003840 con 0x7fe33806d8c0 2024-03-29T11:43:52.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560576062 0 0) 0x7fe3380af4d0 con 0x7fe3380afc40 2024-03-29T11:43:52.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe328003b80 con 0x7fe3380afc40 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667915770 0 0) 0x7fe328003520 con 0x7fe3380aad20 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 >> v1:172.21.15.204:6789/0 conn(0x7fe33806d8c0 legacy=0x7fe3380a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 >> v1:172.21.15.22:6789/0 conn(0x7fe3380afc40 legacy=0x7fe338141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 1 monclient: found mon.noname-b 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380aba30 con 0x7fe3380aad20 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:52.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe32c0015f0 con 0x7fe3380aad20 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe3377fe640 10 monclient: dump: 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.182 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.183 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.183 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:52.183 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:52.183 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:52.183 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:52.183 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:52.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe33da99640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:43:52.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.176+0000 7fe33da99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:52.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.177+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe32c001750 con 0x7fe3380aad20 2024-03-29T11:43:52.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.177+0000 7fe3377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:52.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.177+0000 7fe3377fe640 1 -- 172.21.15.22:0/448995537 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe32c001ba0 con 0x7fe3380aad20 2024-03-29T11:43:52.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.177+0000 7fe3377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.177+0000 7fe3377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:52.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.177+0000 7fe3377fe640 10 monclient: dump: 2024-03-29T11:43:52.185 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.177+0000 7fe33da99640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.177+0000 7fe33da99640 4 set_mon_vals no callback set 2024-03-29T11:43:52.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.178+0000 7fe33da99640 10 monclient: shutdown 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.178+0000 7fe33da99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.178+0000 7fe33da99640 1 -- 172.21.15.22:0/448995537 >> v1:172.21.15.22:6790/0 conn(0x7fe3380aad20 legacy=0x7fe3380ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.179+0000 7fe33da99640 1 -- 172.21.15.22:0/448995537 shutdown_connections 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.179+0000 7fe33da99640 1 -- 172.21.15.22:0/448995537 >> 172.21.15.22:0/448995537 conn(0x7fe3380673e0 msgr2=0x7fe3380b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.179+0000 7fe33da99640 1 -- 172.21.15.22:0/448995537 shutdown_connections 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.179+0000 7fe33da99640 1 -- 172.21.15.22:0/448995537 wait complete. 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 10 monclient: build_initial_monmap 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 10 monclient: monmap: 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:52.187 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:52.188 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:52.188 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:52.188 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:52.188 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:52.188 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:52.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 1 librados: starting msgr at 2024-03-29T11:43:52.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 1 librados: starting objecter 2024-03-29T11:43:52.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 1 Processor -- start 2024-03-29T11:43:52.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 1 -- start start 2024-03-29T11:43:52.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 1 librados: setting wanted keys 2024-03-29T11:43:52.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 1 librados: calling monclient init 2024-03-29T11:43:52.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.180+0000 7fe33da99640 10 monclient: init 2024-03-29T11:43:52.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:52.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:52.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 10 monclient(hunting): picked mon.noname-b con 0x7fe33806d8c0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 10 monclient(hunting): picked mon.noname-a con 0x7fe3380aad20 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 10 monclient(hunting): picked mon.noname-c con 0x7fe3380afc40 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe338071cc0 con 0x7fe3380aad20 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33812cf20 con 0x7fe33806d8c0 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3381e4b30 con 0x7fe3380afc40 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.181+0000 7fe33da99640 10 monclient(hunting): authenticate will time out at 1482.458986s 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.184+0000 7fe33d298640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe3380afc40 0x7fe3381dac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42584/0 (socket says 172.21.15.22:42584) 2024-03-29T11:43:52.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.184+0000 7fe33d298640 1 -- 172.21.15.22:0/1858400208 learned_addr learned my addr 172.21.15.22:0/1858400208 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:52.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.188+0000 7fe3357fa640 10 client.?.objecter ms_handle_connect 0x7fe3380afc40 2024-03-29T11:43:52.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.188+0000 7fe3357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:52.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.188+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852599829 0 0) 0x7fe3381e4b30 con 0x7fe3380afc40 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.188+0000 7fe3357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.188+0000 7fe3357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.188+0000 7fe3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.188+0000 7fe3357fa640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe31c004eb0 con 0x7fe3380afc40 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 10 client.?.objecter ms_handle_connect 0x7fe3380aad20 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 10 client.?.objecter ms_handle_connect 0x7fe33806d8c0 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:52.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12936769 0 0) 0x7fe33812cf20 con 0x7fe33806d8c0 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.189+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3381e4b30 con 0x7fe33806d8c0 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010251449 0 0) 0x7fe338071cc0 con 0x7fe3380aad20 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:43:52.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33812cf20 con 0x7fe3380aad20 2024-03-29T11:43:52.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279193528 0 0) 0x7fe31c004eb0 con 0x7fe3380afc40 2024-03-29T11:43:52.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe338071cc0 con 0x7fe3380afc40 2024-03-29T11:43:52.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe330001cb0 con 0x7fe3380afc40 2024-03-29T11:43:52.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 10 monclient(hunting): dump: 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.190+0000 7fe3357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:52.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.191+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042088192 0 0) 0x7fe3381e4b30 con 0x7fe33806d8c0 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.191+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe31c004eb0 con 0x7fe33806d8c0 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.191+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe32c001a60 con 0x7fe33806d8c0 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.191+0000 7fe3357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.191+0000 7fe3357fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.191+0000 7fe3357fa640 10 monclient(hunting): dump: 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:52.201 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3836190417 0 0) 0x7fe338071cc0 con 0x7fe3380afc40 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 >> v1:172.21.15.204:6789/0 conn(0x7fe33806d8c0 legacy=0x7fe3380715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 >> v1:172.21.15.22:6789/0 conn(0x7fe3380aad20 legacy=0x7fe3381e4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 1 monclient: found mon.c 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:52.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33812d170 con 0x7fe3380afc40 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.194+0000 7fe3357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 10 monclient: _renew_subs 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3381e3c20 con 0x7fe3380afc40 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 10 monclient: _renew_subs 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3381db390 con 0x7fe3380afc40 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.195+0000 7fe33da99640 1 librados: init done 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.197+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe330003990 con 0x7fe3380afc40 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.197+0000 7fe3357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.197+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fe330003d60 con 0x7fe3380afc40 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.197+0000 7fe3357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.197+0000 7fe3357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.197+0000 7fe3357fa640 10 monclient: dump: 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:52.205 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.205 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:52.205 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.205 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:52.205 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:52.205 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:52.205 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:52.205 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:52.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.199+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fe330003ec0 con 0x7fe3380afc40 2024-03-29T11:43:52.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.199+0000 7fe336ffd640 4 set_mon_vals no callback set 2024-03-29T11:43:52.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7fe330084dd0 con 0x7fe3380afc40 2024-03-29T11:43:52.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe3357fa640 10 client.4610.objecter ms_dispatch 0x7fe338000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:52.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe3357fa640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:52.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe3357fa640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:52.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe3357fa640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T11:43:52.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe33da99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe33da99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe33da99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.200+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe318005080 con 0x7fe3380afc40 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.201+0000 7fe33ca97640 10 monclient: get_auth_request con 0x7fe31c06e040 auth_method 0 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.201+0000 7fe3357fa640 10 client.4610.objecter ms_handle_connect 0x7fe31c06e040 2024-03-29T11:43:52.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.212+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3300506b0 con 0x7fe3380afc40 2024-03-29T11:43:52.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.212+0000 7fe3357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.212+0000 7fe3357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:52.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.558+0000 7fe33da99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.558+0000 7fe33da99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.558+0000 7fe33da99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:43:52.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.558+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe318005330 con 0x7fe3380afc40 2024-03-29T11:43:52.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.560+0000 7fe3357fa640 1 -- 172.21.15.22:0/1858400208 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7fe3300525e0 con 0x7fe3380afc40 2024-03-29T11:43:52.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.560+0000 7fe3357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.560+0000 7fe3357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:52.566 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:52.566 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:52.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.564+0000 7fe33da99640 10 librados: watch_flush enter 2024-03-29T11:43:52.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 10 librados: watch_flush exit 2024-03-29T11:43:52.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T11:43:52.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 10 client.4610.objecter successfully canceled tick 2024-03-29T11:43:52.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 >> v1:172.21.15.204:6816/3848099522 conn(0x7fe31c06e040 legacy=0x7fe31c070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 10 monclient: shutdown 2024-03-29T11:43:52.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:52.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 >> v1:172.21.15.22:6790/0 conn(0x7fe3380afc40 legacy=0x7fe3381dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 shutdown_connections 2024-03-29T11:43:52.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.565+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 >> 172.21.15.22:0/1858400208 conn(0x7fe3380673e0 msgr2=0x7fe3381417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:52.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.566+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 shutdown_connections 2024-03-29T11:43:52.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.566+0000 7fe33da99640 1 -- 172.21.15.22:0/1858400208 wait complete. 2024-03-29T11:43:52.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:52.566+0000 7fe33da99640 1 librados: shutdown 2024-03-29T11:43:52.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:59.397 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.529+0000 7ff20a806640 10 monclient: get_monmap_and_config 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.529+0000 7ff20a806640 10 monclient: build_initial_monmap 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.530+0000 7ff20a806640 10 monclient: monmap: 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:43:59.531260+0000 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:43:59.531260+0000 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.530+0000 7ff20a806640 1 Processor -- start 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.530+0000 7ff20a806640 1 -- start start 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.530+0000 7ff20a806640 10 monclient: init 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 10 monclient(hunting): picked mon.noname-c con 0x7ff2041457f0 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 10 monclient(hunting): picked mon.noname-b con 0x7ff20413f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 10 monclient(hunting): picked mon.noname-a con 0x7ff2040a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2040aaef0 con 0x7ff2040a9d70 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff204145bd0 con 0x7ff2041457f0 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff204145eb0 con 0x7ff20413f650 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.531+0000 7ff20a806640 10 monclient(hunting): authenticate will time out at 1489.808586s 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.532+0000 7ff2037fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff20413f650 0x7ff2040b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58232/0 (socket says 172.21.15.22:58232) 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.532+0000 7ff2037fe640 1 -- 172.21.15.22:0/612875769 learned_addr learned my addr 172.21.15.22:0/612875769 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.533+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485718666 0 0) 0x7ff204145eb0 con 0x7ff20413f650 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.533+0000 7ff202ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.533+0000 7ff202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.533+0000 7ff202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.533+0000 7ff202ffd640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.533+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ec003520 con 0x7ff20413f650 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.533+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 743301682 0 0) 0x7ff1ec003520 con 0x7ff20413f650 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.533+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 >> v1:172.21.15.204:6789/0 conn(0x7ff2041457f0 legacy=0x7ff20413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:59.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 >> v1:172.21.15.22:6789/0 conn(0x7ff2040a9d70 legacy=0x7ff20414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:59.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 1 monclient: found mon.noname-b 2024-03-29T11:43:59.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:59.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:43:59.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204146340 con 0x7ff20413f650 2024-03-29T11:43:59.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff1f8001690 con 0x7ff20413f650 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 10 monclient: dump: 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:59.543 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff20a806640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff20a806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f8001860 con 0x7ff20413f650 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff202ffd640 1 -- 172.21.15.22:0/612875769 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff1f8001cb0 con 0x7ff20413f650 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff20a806640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.535+0000 7ff20a806640 4 set_mon_vals no callback set 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.536+0000 7ff202ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.536+0000 7ff202ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.536+0000 7ff202ffd640 10 monclient: dump: 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.544 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.537+0000 7ff20a806640 10 monclient: shutdown 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.537+0000 7ff20a806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.537+0000 7ff20a806640 1 -- 172.21.15.22:0/612875769 >> v1:172.21.15.22:6790/0 conn(0x7ff20413f650 legacy=0x7ff2040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.537+0000 7ff20a806640 1 -- 172.21.15.22:0/612875769 shutdown_connections 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.537+0000 7ff20a806640 1 -- 172.21.15.22:0/612875769 >> 172.21.15.22:0/612875769 conn(0x7ff204066a00 msgr2=0x7ff204066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.537+0000 7ff20a806640 1 -- 172.21.15.22:0/612875769 shutdown_connections 2024-03-29T11:43:59.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.538+0000 7ff20a806640 1 -- 172.21.15.22:0/612875769 wait complete. 2024-03-29T11:43:59.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.539+0000 7ff20a806640 10 monclient: build_initial_monmap 2024-03-29T11:43:59.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.539+0000 7ff20a806640 10 monclient: monmap: 2024-03-29T11:43:59.552 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:43:59.552 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:59.552 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:43:59.552 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:43:59.552 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.539+0000 7ff20a806640 1 librados: starting msgr at 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.539+0000 7ff20a806640 1 librados: starting objecter 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.539+0000 7ff20a806640 1 Processor -- start 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.540+0000 7ff20a806640 1 -- start start 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.540+0000 7ff20a806640 1 librados: setting wanted keys 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.540+0000 7ff20a806640 1 librados: calling monclient init 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.540+0000 7ff20a806640 10 monclient: init 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:59.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 10 monclient(hunting): picked mon.noname-a con 0x7ff2040a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 10 monclient(hunting): picked mon.noname-b con 0x7ff20413f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 10 monclient(hunting): picked mon.noname-c con 0x7ff2041457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20406fc70 con 0x7ff2040a9d70 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20406ff50 con 0x7ff20413f650 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff204142aa0 con 0x7ff2041457f0 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff20a806640 10 monclient(hunting): authenticate will time out at 1489.818416s 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff203fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff2040a9d70 0x7ff2041400b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:43860/0 (socket says 172.21.15.22:43860) 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff203fff640 1 -- 172.21.15.22:0/2411081147 learned_addr learned my addr 172.21.15.22:0/2411081147 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 10 client.?.objecter ms_handle_connect 0x7ff2040a9d70 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:59.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605994401 0 0) 0x7ff20406fc70 con 0x7ff2040a9d70 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f0004eb0 con 0x7ff2040a9d70 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 10 client.?.objecter ms_handle_connect 0x7ff2041457f0 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.541+0000 7ff200ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547418209 0 0) 0x7ff204142aa0 con 0x7ff2041457f0 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff20406fc70 con 0x7ff2041457f0 2024-03-29T11:43:59.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3442768172 0 0) 0x7ff1f0004eb0 con 0x7ff2040a9d70 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff204142aa0 con 0x7ff2040a9d70 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff1f40017d0 con 0x7ff2040a9d70 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:59.556 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908967645 0 0) 0x7ff20406fc70 con 0x7ff2041457f0 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f0004eb0 con 0x7ff2041457f0 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff1fc001b20 con 0x7ff2041457f0 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.557 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1374389631 0 0) 0x7ff204142aa0 con 0x7ff2040a9d70 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 >> v1:172.21.15.22:6790/0 conn(0x7ff2041457f0 legacy=0x7ff2041e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.542+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 >> v1:172.21.15.204:6789/0 conn(0x7ff20413f650 legacy=0x7ff2040706e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff200ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff200ff9640 1 monclient: found mon.a 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff200ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff200ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204138b20 con 0x7ff2040a9d70 2024-03-29T11:43:59.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff200ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff200ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff200ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff20a806640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff20a806640 10 monclient: _renew_subs 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff20a806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.544+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff204071d30 con 0x7ff2040a9d70 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.545+0000 7ff20a806640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.545+0000 7ff20a806640 10 monclient: _renew_subs 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.545+0000 7ff20a806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.545+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff204070fd0 con 0x7ff2040a9d70 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.545+0000 7ff20a806640 1 librados: init done 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.546+0000 7ff20a806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.546+0000 7ff20a806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.546+0000 7ff20a806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.546+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff204142c30 con 0x7ff2040a9d70 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.546+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f4001ce0 con 0x7ff2040a9d70 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.546+0000 7ff200ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.546+0000 7ff2027fc640 4 set_mon_vals no callback set 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.547+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff1f4004a00 con 0x7ff2040a9d70 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.547+0000 7ff200ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.548+0000 7ff200ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.548+0000 7ff200ff9640 10 monclient: dump: 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:43:59.560 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:43:59.561 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:43:59.561 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:43:59.561 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:43:59.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.549+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7ff1f401aa10 con 0x7ff2040a9d70 2024-03-29T11:43:59.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.549+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7ff1f40849d0 con 0x7ff2040a9d70 2024-03-29T11:43:59.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.549+0000 7ff200ff9640 10 client.4575.objecter ms_dispatch 0x7ff204000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:59.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.549+0000 7ff200ff9640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:59.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.549+0000 7ff200ff9640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:59.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.550+0000 7ff200ff9640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T11:43:59.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.550+0000 7ff2037fe640 10 monclient: get_auth_request con 0x7ff1f006de80 auth_method 0 2024-03-29T11:43:59.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.550+0000 7ff200ff9640 10 client.4575.objecter ms_handle_connect 0x7ff1f006de80 2024-03-29T11:43:59.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.557+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1f40502b0 con 0x7ff2040a9d70 2024-03-29T11:43:59.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.557+0000 7ff200ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.557+0000 7ff200ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:59.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.873+0000 7ff20a806640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:59.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.873+0000 7ff20a806640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:59.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.873+0000 7ff20a806640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:43:59.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.873+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff204072080 con 0x7ff2040a9d70 2024-03-29T11:43:59.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.874+0000 7ff200ff9640 1 -- 172.21.15.22:0/2411081147 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7ff1f40521e0 con 0x7ff2040a9d70 2024-03-29T11:43:59.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.874+0000 7ff200ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:59.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.874+0000 7ff200ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:59.880 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:43:59.880 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 10 librados: watch_flush enter 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 10 librados: watch_flush exit 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 10 client.4575.objecter successfully canceled tick 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 >> v1:172.21.15.204:6816/3848099522 conn(0x7ff1f006de80 legacy=0x7ff1f0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 10 monclient: shutdown 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 >> v1:172.21.15.22:6789/0 conn(0x7ff2040a9d70 legacy=0x7ff2041400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 shutdown_connections 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.877+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 >> 172.21.15.22:0/2411081147 conn(0x7ff204066a00 msgr2=0x7ff2040b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:59.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.878+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 shutdown_connections 2024-03-29T11:43:59.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.878+0000 7ff20a806640 1 -- 172.21.15.22:0/2411081147 wait complete. 2024-03-29T11:43:59.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:43:59.878+0000 7ff20a806640 1 librados: shutdown 2024-03-29T11:43:59.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:06.703 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:06.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.854+0000 7f5a60c16640 10 monclient: get_monmap_and_config 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.854+0000 7f5a60c16640 10 monclient: build_initial_monmap 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.854+0000 7f5a60c16640 10 monclient: monmap: 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:44:06.856022+0000 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:44:06.856022+0000 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:44:06.859 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.855+0000 7f5a60c16640 1 Processor -- start 2024-03-29T11:44:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.855+0000 7f5a60c16640 1 -- start start 2024-03-29T11:44:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.855+0000 7f5a60c16640 10 monclient: init 2024-03-29T11:44:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 10 monclient(hunting): picked mon.noname-a con 0x7f5a5c1457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 10 monclient(hunting): picked mon.noname-c con 0x7f5a5c13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 10 monclient(hunting): picked mon.noname-b con 0x7f5a5c0a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c0aaef0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c145bd0 con 0x7f5a5c13f650 2024-03-29T11:44:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c145eb0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:06.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a60c16640 10 monclient(hunting): authenticate will time out at 1497.134450s 2024-03-29T11:44:06.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a5ad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5a5c0a9d70 0x7f5a5c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58380/0 (socket says 172.21.15.22:58380) 2024-03-29T11:44:06.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.857+0000 7f5a5ad9d640 1 -- 172.21.15.22:0/2611718604 learned_addr learned my addr 172.21.15.22:0/2611718604 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754300366 0 0) 0x7f5a5c0aaef0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:44:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a44003520 con 0x7f5a5c1457f0 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018655517 0 0) 0x7f5a5c145eb0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a440048a0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3175172140 0 0) 0x7f5a44003520 con 0x7f5a5c1457f0 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 >> v1:172.21.15.22:6790/0 conn(0x7f5a5c0a9d70 legacy=0x7f5a5c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 >> v1:172.21.15.204:6789/0 conn(0x7f5a5c13f650 legacy=0x7f5a5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient: _finish_hunting 0 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 monclient: found mon.noname-a 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c146340 con 0x7f5a5c1457f0 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient: _finish_auth 0 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 10 monclient: _check_auth_tickets 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a60c16640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a60c16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.858+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5a50001670 con 0x7f5a5c1457f0 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 10 monclient: dump: 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.867 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a60c16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a500018c0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a60c16640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a60c16640 4 set_mon_vals no callback set 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 1 -- 172.21.15.22:0/2611718604 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5a50001c50 con 0x7f5a5c1457f0 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.859+0000 7f5a5959a640 10 monclient: dump: 2024-03-29T11:44:06.868 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.860+0000 7f5a60c16640 10 monclient: shutdown 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.860+0000 7f5a60c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.860+0000 7f5a60c16640 1 -- 172.21.15.22:0/2611718604 >> v1:172.21.15.22:6789/0 conn(0x7f5a5c1457f0 legacy=0x7f5a5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.860+0000 7f5a60c16640 1 -- 172.21.15.22:0/2611718604 shutdown_connections 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.860+0000 7f5a60c16640 1 -- 172.21.15.22:0/2611718604 >> 172.21.15.22:0/2611718604 conn(0x7f5a5c066a00 msgr2=0x7f5a5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.861+0000 7f5a60c16640 1 -- 172.21.15.22:0/2611718604 shutdown_connections 2024-03-29T11:44:06.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.861+0000 7f5a60c16640 1 -- 172.21.15.22:0/2611718604 wait complete. 2024-03-29T11:44:06.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.862+0000 7f5a60c16640 10 monclient: build_initial_monmap 2024-03-29T11:44:06.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.862+0000 7f5a60c16640 10 monclient: monmap: 2024-03-29T11:44:06.881 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:06.881 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:06.881 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:44:06.881 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.862+0000 7f5a60c16640 1 librados: starting msgr at 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.862+0000 7f5a60c16640 1 librados: starting objecter 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.863+0000 7f5a60c16640 1 Processor -- start 2024-03-29T11:44:06.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.863+0000 7f5a60c16640 1 -- start start 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.863+0000 7f5a60c16640 1 librados: setting wanted keys 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.863+0000 7f5a60c16640 1 librados: calling monclient init 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.863+0000 7f5a60c16640 10 monclient: init 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.864+0000 7f5a60c16640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.864+0000 7f5a60c16640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.864+0000 7f5a60c16640 10 monclient(hunting): picked mon.noname-a con 0x7f5a5c0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.864+0000 7f5a5a59c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f5a5c0a9d70 0x7f5a5c1dbed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:39418/0 (socket says 172.21.15.22:39418) 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.864+0000 7f5a5a59c640 1 -- 172.21.15.22:0/3607145229 learned_addr learned my addr 172.21.15.22:0/3607145229 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.864+0000 7f5a4affd640 10 client.?.objecter ms_handle_connect 0x7f5a5c0a9d70 2024-03-29T11:44:06.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.864+0000 7f5a4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a60c16640 10 monclient(hunting): picked mon.noname-b con 0x7f5a5c13f650 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a60c16640 10 monclient(hunting): picked mon.noname-c con 0x7f5a5c1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a4affd640 10 client.?.objecter ms_handle_connect 0x7f5a5c13f650 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c1383c0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a4affd640 10 client.?.objecter ms_handle_connect 0x7f5a5c1457f0 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a4affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c06fbc0 con 0x7f5a5c13f650 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.865+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954539445 0 0) 0x7f5a5c1383c0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c1db9b0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a60c16640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a60c16640 10 monclient(hunting): authenticate will time out at 1497.143182s 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c004eb0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598522295 0 0) 0x7f5a5c06fbc0 con 0x7f5a5c13f650 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c1383c0 con 0x7f5a5c13f650 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839633241 0 0) 0x7f5a5c1db9b0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:06.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.866+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c06fbc0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1420948551 0 0) 0x7f5a3c004eb0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a5c1db9b0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5a50001c50 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient(hunting): dump: 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.886 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3479653274 0 0) 0x7f5a5c06fbc0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a3c004eb0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5a54001cb0 con 0x7f5a5c1457f0 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:06.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient(hunting): dump: 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:06.888 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 139003425 0 0) 0x7f5a5c1db9b0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 >> v1:172.21.15.22:6790/0 conn(0x7f5a5c1457f0 legacy=0x7f5a5c137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 >> v1:172.21.15.204:6789/0 conn(0x7f5a5c13f650 legacy=0x7f5a5c1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 monclient: found mon.a 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c138a90 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient: _finish_auth 0 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 10 monclient: _renew_subs 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c06de90 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 10 monclient: _renew_subs 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:06.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c1dc7c0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.867+0000 7f5a60c16640 1 librados: init done 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.868+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a500038a0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.868+0000 7f5a4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.868+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f5a50003cb0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.868+0000 7f5a4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.868+0000 7f5a4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.868+0000 7f5a4affd640 10 monclient: dump: 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:06.891 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.868+0000 7f5a4bfff640 4 set_mon_vals no callback set 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.870+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f5a50003e10 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.870+0000 7f5a60c16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.870+0000 7f5a60c16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.870+0000 7f5a60c16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.870+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5c138c80 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.870+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f5a50084bb0 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.871+0000 7f5a4affd640 10 client.4581.objecter ms_dispatch 0x7f5a5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:06.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.871+0000 7f5a4affd640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:06.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.871+0000 7f5a4affd640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:06.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.871+0000 7f5a4affd640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-29T11:44:06.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.871+0000 7f5a59d9b640 10 monclient: get_auth_request con 0x7f5a3c06dcd0 auth_method 0 2024-03-29T11:44:06.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.871+0000 7f5a4affd640 10 client.4581.objecter ms_handle_connect 0x7f5a3c06dcd0 2024-03-29T11:44:06.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.879+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a50050490 con 0x7f5a5c0a9d70 2024-03-29T11:44:06.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.879+0000 7f5a4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:06.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:06.879+0000 7f5a4affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:07.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.274+0000 7f5a60c16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.274+0000 7f5a60c16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.274+0000 7f5a60c16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:07.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.274+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a5c06fa20 con 0x7f5a5c0a9d70 2024-03-29T11:44:07.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.275+0000 7f5a4affd640 1 -- 172.21.15.22:0/3607145229 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f5a5c06fa20 con 0x7f5a5c0a9d70 2024-03-29T11:44:07.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.275+0000 7f5a4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.275+0000 7f5a4affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:07.280 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:44:07.280 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:07.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.278+0000 7f5a60c16640 10 librados: watch_flush enter 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.278+0000 7f5a60c16640 10 librados: watch_flush exit 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.278+0000 7f5a60c16640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.278+0000 7f5a60c16640 10 client.4581.objecter successfully canceled tick 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 >> v1:172.21.15.204:6816/3848099522 conn(0x7f5a3c06dcd0 legacy=0x7f5a3c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 10 monclient: shutdown 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 >> v1:172.21.15.22:6789/0 conn(0x7f5a5c0a9d70 legacy=0x7f5a5c1dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 shutdown_connections 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 >> 172.21.15.22:0/3607145229 conn(0x7f5a5c066a00 msgr2=0x7f5a5c13e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:07.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 shutdown_connections 2024-03-29T11:44:07.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 1 -- 172.21.15.22:0/3607145229 wait complete. 2024-03-29T11:44:07.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:07.279+0000 7f5a60c16640 1 librados: shutdown 2024-03-29T11:44:07.298 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:14.105 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:14.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.248+0000 7f616f375640 10 monclient: get_monmap_and_config 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.248+0000 7f616f375640 10 monclient: build_initial_monmap 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.248+0000 7f616f375640 10 monclient: monmap: 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:44:14.250000+0000 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:44:14.250000+0000 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:14.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.251+0000 7f616f375640 1 Processor -- start 2024-03-29T11:44:14.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.251+0000 7f616f375640 1 -- start start 2024-03-29T11:44:14.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.251+0000 7f616f375640 10 monclient: init 2024-03-29T11:44:14.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:14.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:14.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 10 monclient(hunting): picked mon.noname-b con 0x7f6160034850 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:14.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 10 monclient(hunting): picked mon.noname-c con 0x7f616002a640 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:14.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 10 monclient(hunting): picked mon.noname-a con 0x7f61600327d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:14.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616002c820 con 0x7f61600327d0 2024-03-29T11:44:14.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616002a120 con 0x7f616002a640 2024-03-29T11:44:14.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61600c75b0 con 0x7f6160034850 2024-03-29T11:44:14.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:14.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616f375640 10 monclient(hunting): authenticate will time out at 1504.529499s 2024-03-29T11:44:14.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616e373640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6160034850 0x7f61600c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50614/0 (socket says 172.21.15.22:50614) 2024-03-29T11:44:14.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.252+0000 7f616e373640 1 -- 172.21.15.22:0/12590944 learned_addr learned my addr 172.21.15.22:0/12590944 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:14.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.255+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463168796 0 0) 0x7f61600c75b0 con 0x7f6160034850 2024-03-29T11:44:14.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.255+0000 7f616d371640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.255+0000 7f616d371640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.255+0000 7f616d371640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.255+0000 7f616d371640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.255+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6154003520 con 0x7f6160034850 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1255564925 0 0) 0x7f6154003520 con 0x7f6160034850 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 >> v1:172.21.15.204:6789/0 conn(0x7f616002a640 legacy=0x7f61600320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 >> v1:172.21.15.22:6789/0 conn(0x7f61600327d0 legacy=0x7f61600cf240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: _finish_hunting 0 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 1 monclient: found mon.noname-b 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6160033110 con 0x7f6160034850 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: _finish_auth 0 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: _check_auth_tickets 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:14.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f6158001690 con 0x7f6160034850 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616f375640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616f375640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: dump: 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:14.266 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6158001920 con 0x7f6160034850 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 1 -- 172.21.15.22:0/12590944 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f6158001cb0 con 0x7f6160034850 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616d371640 10 monclient: dump: 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:14.267 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616f375640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.256+0000 7f616f375640 4 set_mon_vals no callback set 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.257+0000 7f616f375640 10 monclient: shutdown 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.257+0000 7f616f375640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.257+0000 7f616f375640 1 -- 172.21.15.22:0/12590944 >> v1:172.21.15.22:6790/0 conn(0x7f6160034850 legacy=0x7f61600c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.258+0000 7f616f375640 1 -- 172.21.15.22:0/12590944 shutdown_connections 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.258+0000 7f616f375640 1 -- 172.21.15.22:0/12590944 >> 172.21.15.22:0/12590944 conn(0x7f61600058f0 msgr2=0x7f6160005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.258+0000 7f616f375640 1 -- 172.21.15.22:0/12590944 shutdown_connections 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.258+0000 7f616f375640 1 -- 172.21.15.22:0/12590944 wait complete. 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 10 monclient: build_initial_monmap 2024-03-29T11:44:14.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 10 monclient: monmap: 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 1 librados: starting msgr at 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 1 librados: starting objecter 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 1 Processor -- start 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 1 -- start start 2024-03-29T11:44:14.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 1 librados: setting wanted keys 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 1 librados: calling monclient init 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.259+0000 7f616f375640 10 monclient: init 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 10 monclient(hunting): picked mon.noname-b con 0x7f616002a640 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 10 monclient(hunting): picked mon.noname-a con 0x7f61600327d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 10 monclient(hunting): picked mon.noname-c con 0x7f6160034850 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616002cde0 con 0x7f61600327d0 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616002d0c0 con 0x7f616002a640 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616002d3a0 con 0x7f6160034850 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:14.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.260+0000 7f616f375640 10 monclient(hunting): authenticate will time out at 1504.537871s 2024-03-29T11:44:14.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f616eb74640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6160034850 0x7f6160184f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:51702/0 (socket says 172.21.15.22:51702) 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f616eb74640 1 -- 172.21.15.22:0/1593017116 learned_addr learned my addr 172.21.15.22:0/1593017116 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 client.?.objecter ms_handle_connect 0x7f6160034850 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919384045 0 0) 0x7f616002d3a0 con 0x7f6160034850 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f614c004eb0 con 0x7f6160034850 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 378324693 0 0) 0x7f614c004eb0 con 0x7f6160034850 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f616002d3a0 con 0x7f6160034850 2024-03-29T11:44:14.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f61640018b0 con 0x7f6160034850 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 monclient(hunting): dump: 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.261+0000 7f615effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:14.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.262+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4247598955 0 0) 0x7f616002d3a0 con 0x7f6160034850 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.262+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 >> v1:172.21.15.204:6789/0 conn(0x7f616002a640 legacy=0x7f61600bffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.263+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 >> v1:172.21.15.22:6789/0 conn(0x7f61600327d0 legacy=0x7f616002d7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 1 monclient: found mon.c 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61600bfa80 con 0x7f6160034850 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 10 monclient: _finish_auth 0 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 10 client.?.objecter ms_handle_connect 0x7f61600327d0 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f615effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 10 monclient: _renew_subs 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61600befd0 con 0x7f6160034850 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 10 monclient: _renew_subs 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61600bc1f0 con 0x7f6160034850 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.265+0000 7f616f375640 1 librados: init done 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f616f375640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61640035f0 con 0x7f6160034850 2024-03-29T11:44:14.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f615effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f6164003a00 con 0x7f6160034850 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f615effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f615effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f615effd640 10 monclient: dump: 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:14.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f616f375640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f616f375640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:14.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.267+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6160004cd0 con 0x7f6160034850 2024-03-29T11:44:14.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.269+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f6164003b60 con 0x7f6160034850 2024-03-29T11:44:14.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.269+0000 7f615ffff640 4 set_mon_vals no callback set 2024-03-29T11:44:14.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.269+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f6164084a40 con 0x7f6160034850 2024-03-29T11:44:14.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.269+0000 7f616e373640 10 monclient: get_auth_request con 0x7f614c06dcf0 auth_method 0 2024-03-29T11:44:14.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.270+0000 7f615effd640 10 client.4628.objecter ms_dispatch 0x7f6168000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:14.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.270+0000 7f615effd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:14.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.270+0000 7f615effd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:14.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.270+0000 7f615effd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:44:14.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.270+0000 7f615effd640 10 client.4628.objecter ms_handle_connect 0x7f614c06dcf0 2024-03-29T11:44:14.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.278+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6164050320 con 0x7f6160034850 2024-03-29T11:44:14.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.278+0000 7f615effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.278+0000 7f615effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:14.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.609+0000 7f616f375640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.609+0000 7f616f375640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.609+0000 7f616f375640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:14.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.609+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61600bf160 con 0x7f6160034850 2024-03-29T11:44:14.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.610+0000 7f615effd640 1 -- 172.21.15.22:0/1593017116 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f6164052250 con 0x7f6160034850 2024-03-29T11:44:14.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.610+0000 7f615effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.610+0000 7f615effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:14.615 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:44:14.615 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:14.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.613+0000 7f616f375640 10 librados: watch_flush enter 2024-03-29T11:44:14.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.613+0000 7f616f375640 10 librados: watch_flush exit 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.613+0000 7f616f375640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.613+0000 7f616f375640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 >> v1:172.21.15.204:6816/3848099522 conn(0x7f614c06dcf0 legacy=0x7f614c0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 10 monclient: shutdown 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 >> v1:172.21.15.22:6790/0 conn(0x7f6160034850 legacy=0x7f6160184f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 shutdown_connections 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 >> 172.21.15.22:0/1593017116 conn(0x7f61600058f0 msgr2=0x7f61600c71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 shutdown_connections 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 1 -- 172.21.15.22:0/1593017116 wait complete. 2024-03-29T11:44:14.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:14.614+0000 7f616f375640 1 librados: shutdown 2024-03-29T11:44:14.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:21.441 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:21.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.579+0000 7f28386f4640 10 monclient: get_monmap_and_config 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.579+0000 7f28386f4640 10 monclient: build_initial_monmap 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.579+0000 7f28386f4640 10 monclient: monmap: 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:44:21.580701+0000 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:44:21.580701+0000 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:44:21.584 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.580+0000 7f28386f4640 1 Processor -- start 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.580+0000 7f28386f4640 1 -- start start 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.580+0000 7f28386f4640 10 monclient: init 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 10 monclient(hunting): picked mon.noname-a con 0x7f28300af290 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 10 monclient(hunting): picked mon.noname-c con 0x7f28300ae740 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 10 monclient(hunting): picked mon.noname-b con 0x7f28300af670 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:21.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28300b0980 con 0x7f28300af290 2024-03-29T11:44:21.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28300b39f0 con 0x7f28300ae740 2024-03-29T11:44:21.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28300aebd0 con 0x7f28300af670 2024-03-29T11:44:21.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:21.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f28386f4640 10 monclient(hunting): authenticate will time out at 1511.858720s 2024-03-29T11:44:21.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f2836490640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f28300af290 0x7f28300b4430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:59916/0 (socket says 172.21.15.22:59916) 2024-03-29T11:44:21.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.581+0000 7f2836490640 1 -- 172.21.15.22:0/355114610 learned_addr learned my addr 172.21.15.22:0/355114610 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:21.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651391664 0 0) 0x7f28300b0980 con 0x7f28300af290 2024-03-29T11:44:21.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281c003520 con 0x7f28300af290 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933218879 0 0) 0x7f28300aebd0 con 0x7f28300af670 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28300b0980 con 0x7f28300af670 2024-03-29T11:44:21.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3291967816 0 0) 0x7f28300b0980 con 0x7f28300af670 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.582+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 >> v1:172.21.15.204:6789/0 conn(0x7f28300ae740 legacy=0x7f28301439f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 >> v1:172.21.15.22:6789/0 conn(0x7f28300af290 legacy=0x7f28300b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 10 monclient: _finish_hunting 0 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 1 monclient: found mon.noname-b 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28300b4b40 con 0x7f28300af670 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 10 monclient: _finish_auth 0 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 10 monclient: _check_auth_tickets 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f28280019f0 con 0x7f28300af670 2024-03-29T11:44:21.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:21.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.583+0000 7f283548e640 10 monclient: dump: 2024-03-29T11:44:21.594 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:21.594 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:21.594 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.594 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.594 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f28386f4640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f28386f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2828001b50 con 0x7f28300af670 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f283548e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f283548e640 1 -- 172.21.15.22:0/355114610 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2828001f60 con 0x7f28300af670 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f283548e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f283548e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f283548e640 10 monclient: dump: 2024-03-29T11:44:21.595 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f28386f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.584+0000 7f28386f4640 4 set_mon_vals no callback set 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.585+0000 7f28386f4640 10 monclient: shutdown 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.585+0000 7f28386f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:21.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.585+0000 7f28386f4640 1 -- 172.21.15.22:0/355114610 >> v1:172.21.15.22:6790/0 conn(0x7f28300af670 legacy=0x7f2830151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.585+0000 7f28386f4640 1 -- 172.21.15.22:0/355114610 shutdown_connections 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.585+0000 7f28386f4640 1 -- 172.21.15.22:0/355114610 >> 172.21.15.22:0/355114610 conn(0x7f2830066a00 msgr2=0x7f2830066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.585+0000 7f28386f4640 1 -- 172.21.15.22:0/355114610 shutdown_connections 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.585+0000 7f28386f4640 1 -- 172.21.15.22:0/355114610 wait complete. 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.586+0000 7f28386f4640 10 monclient: build_initial_monmap 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.586+0000 7f28386f4640 10 monclient: monmap: 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:44:21.597 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:44:21.598 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:21.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.587+0000 7f28386f4640 1 librados: starting msgr at 2024-03-29T11:44:21.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.587+0000 7f28386f4640 1 librados: starting objecter 2024-03-29T11:44:21.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.587+0000 7f28386f4640 1 Processor -- start 2024-03-29T11:44:21.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.587+0000 7f28386f4640 1 -- start start 2024-03-29T11:44:21.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.587+0000 7f28386f4640 1 librados: setting wanted keys 2024-03-29T11:44:21.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.587+0000 7f28386f4640 1 librados: calling monclient init 2024-03-29T11:44:21.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.587+0000 7f28386f4640 10 monclient: init 2024-03-29T11:44:21.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.588+0000 7f28386f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:21.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.588+0000 7f28386f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:21.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.588+0000 7f28386f4640 10 monclient(hunting): picked mon.noname-a con 0x7f28300af290 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:21.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.588+0000 7f2836490640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f28300af290 0x7f2830147130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:60230/0 (socket says 172.21.15.22:60230) 2024-03-29T11:44:21.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.588+0000 7f2835c8f640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f28300af670 0x7f2830146220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57928/0 (socket says 172.21.15.22:57928) 2024-03-29T11:44:21.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.588+0000 7f2836490640 1 -- 172.21.15.22:0/2229941880 learned_addr learned my addr 172.21.15.22:0/2229941880 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:21.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f28386f4640 10 monclient(hunting): picked mon.noname-c con 0x7f28300af670 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f28386f4640 10 monclient(hunting): picked mon.noname-b con 0x7f2830142c60 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2830146930 con 0x7f28300af290 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283006ecf0 con 0x7f2830142c60 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28301db340 con 0x7f28300af670 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f28386f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f28386f4640 10 monclient(hunting): authenticate will time out at 1511.866738s 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 client.?.objecter ms_handle_connect 0x7f28300af290 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 client.?.objecter ms_handle_connect 0x7f28300af670 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460548463 0 0) 0x7f28301db340 con 0x7f28300af670 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2814004eb0 con 0x7f28300af670 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909280161 0 0) 0x7f2830146930 con 0x7f28300af290 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.589+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28301db340 con 0x7f28300af290 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 231645268 0 0) 0x7f2814004eb0 con 0x7f28300af670 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2830146930 con 0x7f28300af670 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f28200017a0 con 0x7f28300af670 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:21.606 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4250451692 0 0) 0x7f28301db340 con 0x7f28300af290 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2814004eb0 con 0x7f28300af290 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f282c001a70 con 0x7f28300af290 2024-03-29T11:44:21.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 605331679 0 0) 0x7f2830146930 con 0x7f28300af670 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.590+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 >> v1:172.21.15.204:6789/0 conn(0x7f2830142c60 legacy=0x7f28301e5d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:21.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.591+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 >> v1:172.21.15.22:6789/0 conn(0x7f28300af290 legacy=0x7f2830147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.591+0000 7f2826ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.591+0000 7f2826ffd640 1 monclient: found mon.c 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.591+0000 7f2826ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f2826ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28301db4d0 con 0x7f28300af670 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f2826ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f2826ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f2826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f28386f4640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f28386f4640 10 monclient: _renew_subs 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f28386f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28301db720 con 0x7f28300af670 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f28386f4640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:21.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.592+0000 7f28386f4640 10 monclient: _renew_subs 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.593+0000 7f28386f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.593+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28301e7680 con 0x7f28300af670 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.593+0000 7f28386f4640 1 librados: init done 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.594+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2820001900 con 0x7f28300af670 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.594+0000 7f2826ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.594+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2820004840 con 0x7f28300af670 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.594+0000 7f2826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.594+0000 7f2826ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.594+0000 7f2826ffd640 10 monclient: dump: 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:21.610 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.595+0000 7f28386f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.595+0000 7f28386f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.595+0000 7f28386f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.595+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2830146db0 con 0x7f28300af670 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.595+0000 7f2834c8d640 4 set_mon_vals no callback set 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.595+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f282001a810 con 0x7f28300af670 2024-03-29T11:44:21.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.596+0000 7f2836490640 10 monclient: get_auth_request con 0x7f281406dee0 auth_method 0 2024-03-29T11:44:21.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.596+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f2820084840 con 0x7f28300af670 2024-03-29T11:44:21.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.597+0000 7f2826ffd640 10 client.4634.objecter ms_dispatch 0x7f2830000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:21.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.597+0000 7f2826ffd640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:21.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.597+0000 7f2826ffd640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:21.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.597+0000 7f2826ffd640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:44:21.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.597+0000 7f2826ffd640 10 client.4634.objecter ms_handle_connect 0x7f281406dee0 2024-03-29T11:44:21.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.605+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f282004ff50 con 0x7f28300af670 2024-03-29T11:44:21.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.605+0000 7f2826ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.605+0000 7f2826ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:21.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.924+0000 7f28386f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:21.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.924+0000 7f28386f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:21.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.924+0000 7f28386f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:21.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.924+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2830147840 con 0x7f28300af670 2024-03-29T11:44:21.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.925+0000 7f2826ffd640 1 -- 172.21.15.22:0/2229941880 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f2820052050 con 0x7f28300af670 2024-03-29T11:44:21.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.926+0000 7f2826ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:21.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.926+0000 7f2826ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:21.930 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:44:21.931 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:21.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.928+0000 7f28386f4640 10 librados: watch_flush enter 2024-03-29T11:44:21.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.928+0000 7f28386f4640 10 librados: watch_flush exit 2024-03-29T11:44:21.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.928+0000 7f28386f4640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:44:21.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:44:21.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 >> v1:172.21.15.204:6816/3848099522 conn(0x7f281406dee0 legacy=0x7f28140703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 10 monclient: shutdown 2024-03-29T11:44:21.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:21.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 >> v1:172.21.15.22:6790/0 conn(0x7f28300af670 legacy=0x7f2830146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 shutdown_connections 2024-03-29T11:44:21.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 >> 172.21.15.22:0/2229941880 conn(0x7f2830066a00 msgr2=0x7f28300b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:21.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 shutdown_connections 2024-03-29T11:44:21.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 1 -- 172.21.15.22:0/2229941880 wait complete. 2024-03-29T11:44:21.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:21.929+0000 7f28386f4640 1 librados: shutdown 2024-03-29T11:44:21.949 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:28.756 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.883+0000 7f2ce3f3d640 10 monclient: get_monmap_and_config 2024-03-29T11:44:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.883+0000 7f2ce3f3d640 10 monclient: build_initial_monmap 2024-03-29T11:44:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.883+0000 7f2ce3f3d640 10 monclient: monmap: 2024-03-29T11:44:28.888 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:28.888 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:28.888 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:44:28.885183+0000 2024-03-29T11:44:28.888 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:44:28.885183+0000 2024-03-29T11:44:28.889 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:28.889 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:28.889 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:28.889 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:44:28.889 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:44:28.889 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:28.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.884+0000 7f2ce3f3d640 1 Processor -- start 2024-03-29T11:44:28.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.885+0000 7f2ce3f3d640 1 -- start start 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.885+0000 7f2ce3f3d640 10 monclient: init 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 10 monclient(hunting): picked mon.noname-c con 0x7f2cdc0b4710 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 10 monclient(hunting): picked mon.noname-b con 0x7f2cdc0b3970 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 10 monclient(hunting): picked mon.noname-a con 0x7f2cdc13f390 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0af020 con 0x7f2cdc13f390 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0b0910 con 0x7f2cdc0b4710 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0b0bf0 con 0x7f2cdc0b3970 2024-03-29T11:44:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:28.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce3f3d640 10 monclient(hunting): authenticate will time out at 1519.163161s 2024-03-29T11:44:28.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce14d8640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2cdc0b3970 0x7f2cdc0af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45904/0 (socket says 172.21.15.22:45904) 2024-03-29T11:44:28.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce14d8640 1 -- 172.21.15.22:0/3448863459 learned_addr learned my addr 172.21.15.22:0/3448863459 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:28.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269877256 0 0) 0x7f2cdc0b0bf0 con 0x7f2cdc0b3970 2024-03-29T11:44:28.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce0cd7640 10 monclient(hunting): _negotiate 2024-03-29T11:44:28.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce0cd7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:28.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce0cd7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce0cd7640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:44:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc8003520 con 0x7f2cdc0b3970 2024-03-29T11:44:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069135699 0 0) 0x7f2cc8003520 con 0x7f2cdc0b3970 2024-03-29T11:44:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.886+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 >> v1:172.21.15.204:6789/0 conn(0x7f2cdc0b4710 legacy=0x7f2cdc0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.887+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 >> v1:172.21.15.22:6789/0 conn(0x7f2cdc13f390 legacy=0x7f2cdc13f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.887+0000 7f2ce0cd7640 10 monclient: _finish_hunting 0 2024-03-29T11:44:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.887+0000 7f2ce0cd7640 1 monclient: found mon.noname-b 2024-03-29T11:44:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:44:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc0aeb00 con 0x7f2cdc0b3970 2024-03-29T11:44:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: _finish_auth 0 2024-03-29T11:44:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: _check_auth_tickets 2024-03-29T11:44:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2ccc001690 con 0x7f2cdc0b3970 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: dump: 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:28.899 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce3f3d640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce3f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ccc001920 con 0x7f2cdc0b3970 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 1 -- 172.21.15.22:0/3448863459 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2ccc001d30 con 0x7f2cdc0b3970 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce0cd7640 10 monclient: dump: 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.900 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce3f3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.888+0000 7f2ce3f3d640 4 set_mon_vals no callback set 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.889+0000 7f2ce3f3d640 10 monclient: shutdown 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.889+0000 7f2ce3f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.889+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/3448863459 >> v1:172.21.15.22:6790/0 conn(0x7f2cdc0b3970 legacy=0x7f2cdc0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.889+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/3448863459 shutdown_connections 2024-03-29T11:44:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.889+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/3448863459 >> 172.21.15.22:0/3448863459 conn(0x7f2cdc066a00 msgr2=0x7f2cdc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:28.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.890+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/3448863459 shutdown_connections 2024-03-29T11:44:28.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.890+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/3448863459 wait complete. 2024-03-29T11:44:28.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.890+0000 7f2ce3f3d640 10 monclient: build_initial_monmap 2024-03-29T11:44:28.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.890+0000 7f2ce3f3d640 10 monclient: monmap: 2024-03-29T11:44:28.902 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:28.902 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:28.902 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:44:28.902 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.891+0000 7f2ce3f3d640 1 librados: starting msgr at 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.891+0000 7f2ce3f3d640 1 librados: starting objecter 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.891+0000 7f2ce3f3d640 1 Processor -- start 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.891+0000 7f2ce3f3d640 1 -- start start 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.891+0000 7f2ce3f3d640 1 librados: setting wanted keys 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.891+0000 7f2ce3f3d640 1 librados: calling monclient init 2024-03-29T11:44:28.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.891+0000 7f2ce3f3d640 10 monclient: init 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 10 monclient(hunting): picked mon.noname-b con 0x7f2cdc0b3970 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 10 monclient(hunting): picked mon.noname-c con 0x7f2cdc0b4710 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 10 monclient(hunting): picked mon.noname-a con 0x7f2cdc13f390 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc143220 con 0x7f2cdc13f390 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc06f0f0 con 0x7f2cdc0b3970 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc06f3d0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce3f3d640 10 monclient(hunting): authenticate will time out at 1519.169712s 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce14d8640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2cdc0b4710 0x7f2cdc1410c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47706/0 (socket says 172.21.15.22:47706) 2024-03-29T11:44:28.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.892+0000 7f2ce14d8640 1 -- 172.21.15.22:0/1359703163 learned_addr learned my addr 172.21.15.22:0/1359703163 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 10 client.?.objecter ms_handle_connect 0x7f2cdc0b4710 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581678385 0 0) 0x7f2cdc06f3d0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc0004eb0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.893+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1895577384 0 0) 0x7f2cc0004eb0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cdc06f3d0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2ccc001650 con 0x7f2cdc0b4710 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 10 monclient(hunting): dump: 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2737532992 0 0) 0x7f2cdc06f3d0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 >> v1:172.21.15.204:6789/0 conn(0x7f2cdc0b3970 legacy=0x7f2cdc06fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:28.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.894+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 >> v1:172.21.15.22:6789/0 conn(0x7f2cdc13f390 legacy=0x7f2cdc1e6060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:28.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.896+0000 7f2cd27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:28.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.896+0000 7f2cd27fc640 1 monclient: found mon.c 2024-03-29T11:44:28.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.896+0000 7f2cd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:28.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.896+0000 7f2cd27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:28.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.896+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc06f6b0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.896+0000 7f2cd27fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.896+0000 7f2cd27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.896+0000 7f2cd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 10 monclient: _renew_subs 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc142700 con 0x7f2cdc0b4710 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 10 monclient: _renew_subs 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc06e2e0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2ce3f3d640 1 librados: init done 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ccc004ca0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2cd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f2ccc0050b0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2cd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2cd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2cd27fc640 10 monclient: dump: 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.897+0000 7f2cd3fff640 4 set_mon_vals no callback set 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.898+0000 7f2ce3f3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.898+0000 7f2ce3f3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.899+0000 7f2ce3f3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.899+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc4005080 con 0x7f2cdc0b4710 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.899+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f2ccc005210 con 0x7f2cdc0b4710 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.900+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4289093175 0 0) 0x7f2ccc084cc0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.900+0000 7f2cd27fc640 10 client.4640.objecter ms_dispatch 0x7f2cdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.900+0000 7f2ce1cd9640 10 monclient: get_auth_request con 0x7f2cc006dbe0 auth_method 0 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.900+0000 7f2cd27fc640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.900+0000 7f2cd27fc640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.901+0000 7f2cd27fc640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:44:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.901+0000 7f2cd27fc640 10 client.4640.objecter ms_handle_connect 0x7f2cc006dbe0 2024-03-29T11:44:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.907+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ccc0505a0 con 0x7f2cdc0b4710 2024-03-29T11:44:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.909+0000 7f2cd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:28.909+0000 7f2cd27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.224+0000 7f2ce3f3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.224+0000 7f2ce3f3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.224+0000 7f2ce3f3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.224+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cc4005330 con 0x7f2cdc0b4710 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.225+0000 7f2cd27fc640 1 -- 172.21.15.22:0/1359703163 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 401248453) 0x7f2ccc0524d0 con 0x7f2cdc0b4710 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.225+0000 7f2cd27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.225+0000 7f2cd27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:44:29.230 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:34:32.237924+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:32.237924+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 10 librados: watch_flush enter 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 10 librados: watch_flush exit 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 >> v1:172.21.15.204:6816/3848099522 conn(0x7f2cc006dbe0 legacy=0x7f2cc00700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 10 monclient: shutdown 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 >> v1:172.21.15.22:6790/0 conn(0x7f2cdc0b4710 legacy=0x7f2cdc1410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 shutdown_connections 2024-03-29T11:44:29.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 >> 172.21.15.22:0/1359703163 conn(0x7f2cdc066a00 msgr2=0x7f2cdc0b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 shutdown_connections 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.228+0000 7f2ce3f3d640 1 -- 172.21.15.22:0/1359703163 wait complete. 2024-03-29T11:44:29.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:29.229+0000 7f2ce3f3d640 1 librados: shutdown 2024-03-29T11:44:29.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:36.054 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:36.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.191+0000 7fca8244d640 10 monclient: get_monmap_and_config 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.191+0000 7fca8244d640 10 monclient: build_initial_monmap 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.191+0000 7fca8244d640 10 monclient: monmap: 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:44:36.193203+0000 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:44:36.193203+0000 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:44:36.200 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:44:36.201 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:36.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.192+0000 7fca8244d640 1 Processor -- start 2024-03-29T11:44:36.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.193+0000 7fca8244d640 1 -- start start 2024-03-29T11:44:36.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.193+0000 7fca8244d640 10 monclient: init 2024-03-29T11:44:36.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.193+0000 7fca8244d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:36.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.193+0000 7fca8244d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:36.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.193+0000 7fca8244d640 10 monclient(hunting): picked mon.noname-b con 0x7fca7c0b01a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:36.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.193+0000 7fca8244d640 10 monclient(hunting): picked mon.noname-c con 0x7fca7c0af010 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:36.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.195+0000 7fca8244d640 10 monclient(hunting): picked mon.noname-a con 0x7fca7c0ac5a0 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:36.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.195+0000 7fca8244d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c0ae8a0 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca8244d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c0ab1d0 con 0x7fca7c0af010 2024-03-29T11:44:36.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca809ea640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fca7c0ac5a0 0x7fca7c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48753/0 (socket says 172.21.15.22:48753) 2024-03-29T11:44:36.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca809ea640 1 -- 172.21.15.22:0/414111662 learned_addr learned my addr 172.21.15.22:0/414111662 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:36.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca8244d640 1 -- 172.21.15.22:0/414111662 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c0ab4b0 con 0x7fca7c0b01a0 2024-03-29T11:44:36.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca8244d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:36.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca8244d640 10 monclient(hunting): authenticate will time out at 1526.473128s 2024-03-29T11:44:36.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960033974 0 0) 0x7fca7c0ae8a0 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64003520 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4149840024 0 0) 0x7fca64003520 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 >> v1:172.21.15.22:6790/0 conn(0x7fca7c0b01a0 legacy=0x7fca7c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 >> v1:172.21.15.204:6789/0 conn(0x7fca7c0af010 legacy=0x7fca7c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 1 monclient: found mon.noname-a 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0ab790 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient: _finish_auth 0 2024-03-29T11:44:36.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:36.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:36.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fca74001a10 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca7affd640 10 monclient: dump: 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca8244d640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.196+0000 7fca8244d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.197+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca74001b70 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.197+0000 7fca7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.197+0000 7fca7affd640 1 -- 172.21.15.22:0/414111662 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fca74001fc0 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.197+0000 7fca7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.197+0000 7fca7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.197+0000 7fca7affd640 10 monclient: dump: 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:36.215 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.197+0000 7fca8244d640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.197+0000 7fca8244d640 4 set_mon_vals no callback set 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.198+0000 7fca8244d640 10 monclient: shutdown 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.198+0000 7fca8244d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.198+0000 7fca8244d640 1 -- 172.21.15.22:0/414111662 >> v1:172.21.15.22:6789/0 conn(0x7fca7c0ac5a0 legacy=0x7fca7c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.198+0000 7fca8244d640 1 -- 172.21.15.22:0/414111662 shutdown_connections 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.198+0000 7fca8244d640 1 -- 172.21.15.22:0/414111662 >> 172.21.15.22:0/414111662 conn(0x7fca7c06b660 msgr2=0x7fca7c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.198+0000 7fca8244d640 1 -- 172.21.15.22:0/414111662 shutdown_connections 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.198+0000 7fca8244d640 1 -- 172.21.15.22:0/414111662 wait complete. 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.199+0000 7fca8244d640 10 monclient: build_initial_monmap 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.199+0000 7fca8244d640 10 monclient: monmap: 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.199+0000 7fca8244d640 1 librados: starting msgr at 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.199+0000 7fca8244d640 1 librados: starting objecter 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.199+0000 7fca8244d640 1 Processor -- start 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.200+0000 7fca8244d640 1 -- start start 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.200+0000 7fca8244d640 1 librados: setting wanted keys 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.200+0000 7fca8244d640 1 librados: calling monclient init 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.200+0000 7fca8244d640 10 monclient: init 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.200+0000 7fca8244d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.200+0000 7fca8244d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca8244d640 10 monclient(hunting): picked mon.noname-b con 0x7fca7c0ac5a0 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca8244d640 10 monclient(hunting): picked mon.noname-c con 0x7fca7c0b01a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca8244d640 10 monclient(hunting): picked mon.noname-a con 0x7fca7c06f990 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca8244d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c06fe20 con 0x7fca7c06f990 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca8244d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c14bb90 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca8244d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c14be70 con 0x7fca7c0b01a0 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca8244d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca8244d640 10 monclient(hunting): authenticate will time out at 1526.480241s 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca7b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fca7c0b01a0 0x7fca7c0703c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:54780/0 (socket says 172.21.15.22:54780) 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca7b7fe640 1 -- 172.21.15.22:0/2698525304 learned_addr learned my addr 172.21.15.22:0/2698525304 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca78ff9640 10 client.?.objecter ms_handle_connect 0x7fca7c0b01a0 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408560939 0 0) 0x7fca7c14be70 con 0x7fca7c0b01a0 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca78ff9640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.203+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca68004eb0 con 0x7fca7c0b01a0 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 542614535 0 0) 0x7fca68004eb0 con 0x7fca7c0b01a0 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca7c14be70 con 0x7fca7c0b01a0 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fca70001940 con 0x7fca7c0b01a0 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:36.228 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:36.229 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 10 client.?.objecter ms_handle_connect 0x7fca7c0ac5a0 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.204+0000 7fca78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534094810 0 0) 0x7fca7c14bb90 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca68004eb0 con 0x7fca7c0ac5a0 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2941597887 0 0) 0x7fca7c14be70 con 0x7fca7c0b01a0 2024-03-29T11:44:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 >> v1:172.21.15.204:6789/0 conn(0x7fca7c0ac5a0 legacy=0x7fca7c1340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.205+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 >> v1:172.21.15.22:6789/0 conn(0x7fca7c06f990 legacy=0x7fca7c149560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca78ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca78ff9640 1 monclient: found mon.c 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca78ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c070ad0 con 0x7fca7c0b01a0 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca78ff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca78ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 10 monclient: _renew_subs 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c132470 con 0x7fca7c0b01a0 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 10 monclient: _renew_subs 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c14adf0 con 0x7fca7c0b01a0 2024-03-29T11:44:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.207+0000 7fca8244d640 1 librados: init done 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.208+0000 7fca8244d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.208+0000 7fca8244d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.208+0000 7fca8244d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.208+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c070040 con 0x7fca7c0b01a0 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.210+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca70001c60 con 0x7fca7c0b01a0 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.210+0000 7fca78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.210+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7fca70003a60 con 0x7fca7c0b01a0 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.210+0000 7fca78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.210+0000 7fca78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.210+0000 7fca78ff9640 10 monclient: dump: 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:36.234 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.210+0000 7fca7a7fc640 4 set_mon_vals no callback set 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.212+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7fca70003bc0 con 0x7fca7c0b01a0 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.213+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1368017700 0 0) 0x7fca70084a60 con 0x7fca7c0b01a0 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.213+0000 7fca78ff9640 10 client.4646.objecter ms_dispatch 0x7fca7c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.213+0000 7fca78ff9640 3 client.4646.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.213+0000 7fca78ff9640 3 client.4646.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.213+0000 7fca78ff9640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T11:44:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.213+0000 7fca7bfff640 10 monclient: get_auth_request con 0x7fca6806dac0 auth_method 0 2024-03-29T11:44:36.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.214+0000 7fca78ff9640 10 client.4646.objecter ms_handle_connect 0x7fca6806dac0 2024-03-29T11:44:36.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.218+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca70050340 con 0x7fca7c0b01a0 2024-03-29T11:44:36.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.218+0000 7fca78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.218+0000 7fca78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:36.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.557+0000 7fca8244d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:36.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.557+0000 7fca8244d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:36.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.557+0000 7fca8244d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:36.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.557+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca7c075f10 con 0x7fca7c0b01a0 2024-03-29T11:44:36.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.560+0000 7fca78ff9640 1 -- 172.21.15.22:0/2698525304 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1933163311) 0x7fca70052270 con 0x7fca7c0b01a0 2024-03-29T11:44:36.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.560+0000 7fca78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:36.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.560+0000 7fca78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:36.565 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:44:36.565 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:44:34.925774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:34.925774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.564+0000 7fca8244d640 10 librados: watch_flush enter 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 10 librados: watch_flush exit 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 10 client.4646.objecter successfully canceled tick 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 >> v1:172.21.15.204:6816/3848099522 conn(0x7fca6806dac0 legacy=0x7fca6806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 10 monclient: shutdown 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 >> v1:172.21.15.22:6790/0 conn(0x7fca7c0b01a0 legacy=0x7fca7c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 shutdown_connections 2024-03-29T11:44:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.565+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 >> 172.21.15.22:0/2698525304 conn(0x7fca7c06b660 msgr2=0x7fca7c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:36.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.566+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 shutdown_connections 2024-03-29T11:44:36.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.566+0000 7fca8244d640 1 -- 172.21.15.22:0/2698525304 wait complete. 2024-03-29T11:44:36.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:36.566+0000 7fca8244d640 1 librados: shutdown 2024-03-29T11:44:36.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:43.395 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.541+0000 7f25df7cf640 10 monclient: get_monmap_and_config 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.541+0000 7f25df7cf640 10 monclient: build_initial_monmap 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.541+0000 7f25df7cf640 10 monclient: monmap: 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:44:43.542768+0000 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:44:43.542768+0000 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:43.547 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:43.548 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:43.548 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:44:43.548 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:44:43.548 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:43.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.542+0000 7f25df7cf640 1 Processor -- start 2024-03-29T11:44:43.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.542+0000 7f25df7cf640 1 -- start start 2024-03-29T11:44:43.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.542+0000 7f25df7cf640 10 monclient: init 2024-03-29T11:44:43.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.543+0000 7f25df7cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:43.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.543+0000 7f25df7cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:43.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.543+0000 7f25df7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f25d80a9ac0 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:43.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25df7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f25d813ee90 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:43.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25df7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f25d80af900 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25df7cf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d80afd10 con 0x7f25d813ee90 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25df7cf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d80af6f0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25df7cf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d80aac80 con 0x7f25d80af900 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25df7cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25df7cf640 10 monclient(hunting): authenticate will time out at 1533.821512s 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25ddd6c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f25d80af900 0x7f25d8149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53556/0 (socket says 172.21.15.22:53556) 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.544+0000 7f25ddd6c640 1 -- 172.21.15.22:0/229347349 learned_addr learned my addr 172.21.15.22:0/229347349 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.545+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14865221 0 0) 0x7f25d80aac80 con 0x7f25d80af900 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.545+0000 7f25cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.545+0000 7f25cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.545+0000 7f25cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.545+0000 7f25cffff640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.545+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c4003520 con 0x7f25d80af900 2024-03-29T11:44:43.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.545+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126917505 0 0) 0x7f25c4003520 con 0x7f25d80af900 2024-03-29T11:44:43.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.545+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 >> v1:172.21.15.204:6789/0 conn(0x7f25d80a9ac0 legacy=0x7f25d80adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:43.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.547+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 >> v1:172.21.15.22:6789/0 conn(0x7f25d813ee90 legacy=0x7f25d813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 1 monclient: found mon.noname-b 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d80aaf60 con 0x7f25d80af900 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 10 monclient: _finish_auth 0 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f25c8001690 con 0x7f25d80af900 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:43.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25cffff640 10 monclient: dump: 2024-03-29T11:44:43.558 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:43.558 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25df7cf640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.548+0000 7f25df7cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25c8001860 con 0x7f25d80af900 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25df7cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25df7cf640 4 set_mon_vals no callback set 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25cffff640 1 -- 172.21.15.22:0/229347349 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f25c8001c70 con 0x7f25d80af900 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25cffff640 10 monclient: dump: 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25df7cf640 10 monclient: shutdown 2024-03-29T11:44:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25df7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.549+0000 7f25df7cf640 1 -- 172.21.15.22:0/229347349 >> v1:172.21.15.22:6790/0 conn(0x7f25d80af900 legacy=0x7f25d8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.550+0000 7f25df7cf640 1 -- 172.21.15.22:0/229347349 shutdown_connections 2024-03-29T11:44:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.550+0000 7f25df7cf640 1 -- 172.21.15.22:0/229347349 >> 172.21.15.22:0/229347349 conn(0x7f25d80b35e0 msgr2=0x7f25d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.550+0000 7f25df7cf640 1 -- 172.21.15.22:0/229347349 shutdown_connections 2024-03-29T11:44:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.550+0000 7f25df7cf640 1 -- 172.21.15.22:0/229347349 wait complete. 2024-03-29T11:44:43.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.555+0000 7f25df7cf640 10 monclient: build_initial_monmap 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.555+0000 7f25df7cf640 10 monclient: monmap: 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.555+0000 7f25df7cf640 1 librados: starting msgr at 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.556+0000 7f25df7cf640 1 librados: starting objecter 2024-03-29T11:44:43.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.556+0000 7f25df7cf640 1 Processor -- start 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.556+0000 7f25df7cf640 1 -- start start 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.556+0000 7f25df7cf640 1 librados: setting wanted keys 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.556+0000 7f25df7cf640 1 librados: calling monclient init 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.556+0000 7f25df7cf640 10 monclient: init 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.557+0000 7f25df7cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.557+0000 7f25df7cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.557+0000 7f25df7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f25d80a9ac0 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.558+0000 7f25df7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f25d80af900 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.558+0000 7f25df7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f25d813ee90 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.558+0000 7f25df7cf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d81debc0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.558+0000 7f25df7cf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d812b430 con 0x7f25d813ee90 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.558+0000 7f25df7cf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d813c3f0 con 0x7f25d80af900 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.558+0000 7f25df7cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:43.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.558+0000 7f25df7cf640 10 monclient(hunting): authenticate will time out at 1533.835080s 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.561+0000 7f25dd56b640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f25d80a9ac0 0x7f25d8142050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:39955/0 (socket says 172.21.15.22:39955) 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.561+0000 7f25dd56b640 1 -- 172.21.15.22:0/52637172 learned_addr learned my addr 172.21.15.22:0/52637172 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.562+0000 7f25cdffb640 10 client.?.objecter ms_handle_connect 0x7f25d80a9ac0 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.562+0000 7f25cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.562+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338520279 0 0) 0x7f25d81debc0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.562+0000 7f25cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.562+0000 7f25cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.562+0000 7f25cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.562+0000 7f25cdffb640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:44:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.562+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25bc004eb0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 10 client.?.objecter ms_handle_connect 0x7f25d80af900 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 10 client.?.objecter ms_handle_connect 0x7f25d813ee90 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2372274821 0 0) 0x7f25bc004eb0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25d81debc0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f25d0001bf0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 10 monclient(hunting): dump: 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:43.574 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.563+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357606799 0 0) 0x7f25d813c3f0 con 0x7f25d80af900 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25bc004eb0 con 0x7f25d80af900 2024-03-29T11:44:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 482565250 0 0) 0x7f25d81debc0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 >> v1:172.21.15.22:6790/0 conn(0x7f25d80af900 legacy=0x7f25d81df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 >> v1:172.21.15.204:6789/0 conn(0x7f25d813ee90 legacy=0x7f25d81287e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 1 monclient: found mon.a 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d813c580 con 0x7f25d80a9ac0 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.564+0000 7f25cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 10 monclient: _renew_subs 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d8140630 con 0x7f25d80a9ac0 2024-03-29T11:44:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 10 monclient: _renew_subs 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d813e6f0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25df7cf640 1 librados: init done 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d0002140 con 0x7f25d80a9ac0 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f25d0004c80 con 0x7f25d80a9ac0 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.565+0000 7f25cdffb640 10 monclient: dump: 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:43.577 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.567+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f25d0004de0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.568+0000 7f25ceffd640 4 set_mon_vals no callback set 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.568+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1368017700 0 0) 0x7f25d0084cd0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.568+0000 7f25cdffb640 10 client.4596.objecter ms_dispatch 0x7f25d8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.568+0000 7f25cdffb640 3 client.4596.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.568+0000 7f25cdffb640 3 client.4596.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:43.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.568+0000 7f25cdffb640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-29T11:44:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.569+0000 7f25dcd6a640 10 monclient: get_auth_request con 0x7f25bc06dbc0 auth_method 0 2024-03-29T11:44:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.569+0000 7f25cdffb640 10 client.4596.objecter ms_handle_connect 0x7f25bc06dbc0 2024-03-29T11:44:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.569+0000 7f25df7cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.569+0000 7f25df7cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.569+0000 7f25df7cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.569+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b8005080 con 0x7f25d80a9ac0 2024-03-29T11:44:43.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.577+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25d00505b0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.577+0000 7f25cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:43.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.577+0000 7f25cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:43.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.905+0000 7f25df7cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:43.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.905+0000 7f25df7cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:43.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.905+0000 7f25df7cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:44:43.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.905+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25b8005330 con 0x7f25d80a9ac0 2024-03-29T11:44:43.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.906+0000 7f25cdffb640 1 -- 172.21.15.22:0/52637172 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1933163311) 0x7f25d00524e0 con 0x7f25d80a9ac0 2024-03-29T11:44:43.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.906+0000 7f25cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:43.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.906+0000 7f25cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:43.911 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:44:43.911 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:44:34.925774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:34.925774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:43.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.909+0000 7f25df7cf640 10 librados: watch_flush enter 2024-03-29T11:44:43.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.910+0000 7f25df7cf640 10 librados: watch_flush exit 2024-03-29T11:44:43.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.910+0000 7f25df7cf640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-29T11:44:43.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.910+0000 7f25df7cf640 10 client.4596.objecter successfully canceled tick 2024-03-29T11:44:43.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.910+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 >> v1:172.21.15.204:6816/3848099522 conn(0x7f25bc06dbc0 legacy=0x7f25bc070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.910+0000 7f25df7cf640 10 monclient: shutdown 2024-03-29T11:44:43.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.910+0000 7f25df7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:43.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.910+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 >> v1:172.21.15.22:6789/0 conn(0x7f25d80a9ac0 legacy=0x7f25d8142050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.911+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 shutdown_connections 2024-03-29T11:44:43.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.911+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 >> 172.21.15.22:0/52637172 conn(0x7f25d80b35e0 msgr2=0x7f25d812ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:43.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.914+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 shutdown_connections 2024-03-29T11:44:43.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.914+0000 7f25df7cf640 1 -- 172.21.15.22:0/52637172 wait complete. 2024-03-29T11:44:43.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:43.914+0000 7f25df7cf640 1 librados: shutdown 2024-03-29T11:44:43.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:50.742 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:50.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.882+0000 7f24b9554640 10 monclient: get_monmap_and_config 2024-03-29T11:44:50.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.882+0000 7f24b9554640 10 monclient: build_initial_monmap 2024-03-29T11:44:50.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.882+0000 7f24b9554640 10 monclient: monmap: 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:44:50.884194+0000 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:44:50.884194+0000 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:44:50.888 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:50.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.884+0000 7f24b9554640 1 Processor -- start 2024-03-29T11:44:50.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.884+0000 7f24b9554640 1 -- start start 2024-03-29T11:44:50.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.884+0000 7f24b9554640 10 monclient: init 2024-03-29T11:44:50.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.885+0000 7f24b9554640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:50.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.885+0000 7f24b9554640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:50.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.888+0000 7f24b9554640 10 monclient(hunting): picked mon.noname-a con 0x7f24ac0389e0 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:50.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.888+0000 7f24b3fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f24ac0389e0 0x7f24ac033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:47317/0 (socket says 172.21.15.22:47317) 2024-03-29T11:44:50.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.888+0000 7f24b3fff640 1 -- 172.21.15.22:0/2531882041 learned_addr learned my addr 172.21.15.22:0/2531882041 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:50.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.888+0000 7f24b9554640 10 monclient(hunting): picked mon.noname-b con 0x7f24ac038dc0 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:50.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.888+0000 7f24b9554640 10 monclient(hunting): picked mon.noname-c con 0x7f24ac030660 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:50.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.888+0000 7f24b9554640 1 -- 172.21.15.22:0/2531882041 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac0c54c0 con 0x7f24ac0389e0 2024-03-29T11:44:50.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b9554640 1 -- 172.21.15.22:0/2531882041 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac036430 con 0x7f24ac030660 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b9554640 1 -- 172.21.15.22:0/2531882041 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac039840 con 0x7f24ac038dc0 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630538161 0 0) 0x7f24ac0c54c0 con 0x7f24ac0389e0 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b9554640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b9554640 10 monclient(hunting): authenticate will time out at 1541.166257s 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a4003520 con 0x7f24ac0389e0 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 597691386 0 0) 0x7f24a4003520 con 0x7f24ac0389e0 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 >> v1:172.21.15.22:6790/0 conn(0x7f24ac038dc0 legacy=0x7f24ac0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:50.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.889+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 >> v1:172.21.15.204:6789/0 conn(0x7f24ac030660 legacy=0x7f24ac0d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:50.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:50.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:44:50.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:50.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:44:50.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac035690 con 0x7f24ac0389e0 2024-03-29T11:44:50.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:50.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:50.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f24a8001690 con 0x7f24ac0389e0 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.891+0000 7f24b2ffd640 10 monclient: dump: 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:50.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.892+0000 7f24b9554640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.892+0000 7f24b9554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.892+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a8001860 con 0x7f24ac0389e0 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.892+0000 7f24b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.892+0000 7f24b2ffd640 1 -- 172.21.15.22:0/2531882041 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f24a8001cb0 con 0x7f24ac0389e0 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.892+0000 7f24b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.892+0000 7f24b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.892+0000 7f24b2ffd640 10 monclient: dump: 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:50.900 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:50.901 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:50.901 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:50.901 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.893+0000 7f24b9554640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.893+0000 7f24b9554640 4 set_mon_vals no callback set 2024-03-29T11:44:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.894+0000 7f24b9554640 10 monclient: shutdown 2024-03-29T11:44:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.894+0000 7f24b9554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.894+0000 7f24b9554640 1 -- 172.21.15.22:0/2531882041 >> v1:172.21.15.22:6789/0 conn(0x7f24ac0389e0 legacy=0x7f24ac033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:50.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.894+0000 7f24b9554640 1 -- 172.21.15.22:0/2531882041 shutdown_connections 2024-03-29T11:44:50.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.894+0000 7f24b9554640 1 -- 172.21.15.22:0/2531882041 >> 172.21.15.22:0/2531882041 conn(0x7f24ac005c00 msgr2=0x7f24ac006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.895+0000 7f24b9554640 1 -- 172.21.15.22:0/2531882041 shutdown_connections 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.896+0000 7f24b9554640 1 -- 172.21.15.22:0/2531882041 wait complete. 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.896+0000 7f24b9554640 10 monclient: build_initial_monmap 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.896+0000 7f24b9554640 10 monclient: monmap: 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:44:50.903 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:50.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.897+0000 7f24b9554640 1 librados: starting msgr at 2024-03-29T11:44:50.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.897+0000 7f24b9554640 1 librados: starting objecter 2024-03-29T11:44:50.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.897+0000 7f24b9554640 1 Processor -- start 2024-03-29T11:44:50.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.898+0000 7f24b9554640 1 -- start start 2024-03-29T11:44:50.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.898+0000 7f24b9554640 1 librados: setting wanted keys 2024-03-29T11:44:50.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.898+0000 7f24b9554640 1 librados: calling monclient init 2024-03-29T11:44:50.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.898+0000 7f24b9554640 10 monclient: init 2024-03-29T11:44:50.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.899+0000 7f24b9554640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:50.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.899+0000 7f24b9554640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:50.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.902+0000 7f24b9554640 10 monclient(hunting): picked mon.noname-a con 0x7f24ac030660 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:50.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.902+0000 7f24b3fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f24ac030660 0x7f24ac0c2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:47519/0 (socket says 172.21.15.22:47519) 2024-03-29T11:44:50.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.902+0000 7f24b3fff640 1 -- 172.21.15.22:0/2180863050 learned_addr learned my addr 172.21.15.22:0/2180863050 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:50.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.902+0000 7f24b9554640 10 monclient(hunting): picked mon.noname-b con 0x7f24ac0389e0 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.902+0000 7f24b0ff9640 10 client.?.objecter ms_handle_connect 0x7f24ac030660 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.902+0000 7f24b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.902+0000 7f24b9554640 10 monclient(hunting): picked mon.noname-c con 0x7f24ac0c84d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.902+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac0ca5d0 con 0x7f24ac030660 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b0ff9640 10 client.?.objecter ms_handle_connect 0x7f24ac0c84d0 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac0c9360 con 0x7f24ac0389e0 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac0c8ff0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b9554640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:50.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b9554640 10 monclient(hunting): authenticate will time out at 1541.180566s 2024-03-29T11:44:50.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224290701 0 0) 0x7f24ac0c8ff0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.903+0000 7f24b0ff9640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.904+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2498004ee0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.905+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737468349 0 0) 0x7f24ac0ca5d0 con 0x7f24ac030660 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.905+0000 7f24b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.905+0000 7f24b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.905+0000 7f24b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:50.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.905+0000 7f24b0ff9640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.905+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ac0c8ff0 con 0x7f24ac030660 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034082238 0 0) 0x7f2498004ee0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24ac0ca5d0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 client.?.objecter ms_handle_connect 0x7f24ac0389e0 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f24b404fdd0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:50.918 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:50.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3083885301 0 0) 0x7f24ac0c8ff0 con 0x7f24ac030660 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2498004ee0 con 0x7f24ac030660 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f24a8001b70 con 0x7f24ac030660 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.920 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045718852 0 0) 0x7f24ac0c9360 con 0x7f24ac0389e0 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:50.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.906+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ac0c8ff0 con 0x7f24ac0389e0 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2988230440 0 0) 0x7f24ac0ca5d0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 >> v1:172.21.15.204:6789/0 conn(0x7f24ac0389e0 legacy=0x7f24ac15fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 >> v1:172.21.15.22:6789/0 conn(0x7f24ac030660 legacy=0x7f24ac0c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 1 monclient: found mon.c 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac0c2d80 con 0x7f24ac0c84d0 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:50.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 10 monclient: _renew_subs 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ac0c2fd0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 10 monclient: _renew_subs 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ac0c29f0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.907+0000 7f24b9554640 1 librados: init done 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.909+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24b404f400 con 0x7f24ac0c84d0 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.909+0000 7f24b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.909+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f24b404f730 con 0x7f24ac0c84d0 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.909+0000 7f24b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.909+0000 7f24b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.909+0000 7f24b0ff9640 10 monclient: dump: 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.910+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f24b4067bf0 con 0x7f24ac0c84d0 2024-03-29T11:44:50.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.910+0000 7f24b27fc640 4 set_mon_vals no callback set 2024-03-29T11:44:50.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.911+0000 7f24b9554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:50.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.911+0000 7f24b9554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:50.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.911+0000 7f24b9554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:50.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.911+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2494005080 con 0x7f24ac0c84d0 2024-03-29T11:44:50.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.911+0000 7f24b3fff640 10 monclient: get_auth_request con 0x7f2498076670 auth_method 0 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.913+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1368017700 0 0) 0x7f24b40fe050 con 0x7f24ac0c84d0 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.913+0000 7f24b0ff9640 10 client.4658.objecter ms_dispatch 0x7f24b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.913+0000 7f24b0ff9640 3 client.4658.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.913+0000 7f24b0ff9640 3 client.4658.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.913+0000 7f24b0ff9640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.914+0000 7f24b0ff9640 10 client.4658.objecter ms_handle_connect 0x7f2498076670 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.920+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24b40c9990 con 0x7f24ac0c84d0 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.920+0000 7f24b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:50.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:50.920+0000 7f24b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:51.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.246+0000 7f24b9554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.246+0000 7f24b9554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.246+0000 7f24b9554640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:44:51.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.246+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2494005330 con 0x7f24ac0c84d0 2024-03-29T11:44:51.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.248+0000 7f24b0ff9640 1 -- 172.21.15.22:0/2180863050 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1933163311) 0x7f24b40cb8c0 con 0x7f24ac0c84d0 2024-03-29T11:44:51.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.248+0000 7f24b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.248+0000 7f24b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:51.254 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:44:51.254 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:44:34.925774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:34.925774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:51.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.251+0000 7f24b9554640 10 librados: watch_flush enter 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.251+0000 7f24b9554640 10 librados: watch_flush exit 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.251+0000 7f24b9554640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.251+0000 7f24b9554640 10 client.4658.objecter successfully canceled tick 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.252+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 >> v1:172.21.15.204:6816/3848099522 conn(0x7f2498076670 legacy=0x7f2498078b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.252+0000 7f24b9554640 10 monclient: shutdown 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.252+0000 7f24b9554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.252+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 >> v1:172.21.15.22:6790/0 conn(0x7f24ac0c84d0 legacy=0x7f24ac160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.252+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 shutdown_connections 2024-03-29T11:44:51.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.252+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 >> 172.21.15.22:0/2180863050 conn(0x7f24ac005c00 msgr2=0x7f24ac034af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:51.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.257+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 shutdown_connections 2024-03-29T11:44:51.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.257+0000 7f24b9554640 1 -- 172.21.15.22:0/2180863050 wait complete. 2024-03-29T11:44:51.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:51.257+0000 7f24b9554640 1 librados: shutdown 2024-03-29T11:44:51.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:58.086 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.237+0000 7faa8bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.237+0000 7faa8bfff640 10 monclient: build_initial_monmap 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.237+0000 7faa8bfff640 10 monclient: monmap: 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:44:58.237862+0000 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:44:58.237862+0000 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:58.241 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:58.242 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:58.242 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:44:58.242 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:44:58.242 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:58.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.239+0000 7faa8bfff640 1 Processor -- start 2024-03-29T11:44:58.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.239+0000 7faa8bfff640 1 -- start start 2024-03-29T11:44:58.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.239+0000 7faa8bfff640 10 monclient: init 2024-03-29T11:44:58.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:58.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 10 monclient(hunting): picked mon.noname-c con 0x7faa8c0aad20 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c06d8c0 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c0afc40 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0af4d0 con 0x7faa8c0afc40 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0af7b0 con 0x7faa8c0aad20 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0acab0 con 0x7faa8c06d8c0 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8bfff640 10 monclient(hunting): authenticate will time out at 1548.516217s 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8a7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7faa8c06d8c0 0x7faa8c0ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56006/0 (socket says 172.21.15.22:56006) 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.240+0000 7faa8a7fc640 1 -- 172.21.15.22:0/3433988092 learned_addr learned my addr 172.21.15.22:0/3433988092 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.241+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897308424 0 0) 0x7faa8c0acab0 con 0x7faa8c06d8c0 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.242+0000 7faa89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.242+0000 7faa89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:58.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.242+0000 7faa89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.242+0000 7faa89ffb640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.242+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa7400bd30 con 0x7faa8c06d8c0 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2840387432 0 0) 0x7faa7400bd30 con 0x7faa8c06d8c0 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 >> v1:172.21.15.204:6789/0 conn(0x7faa8c0aad20 legacy=0x7faa8c0a9750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 >> v1:172.21.15.22:6789/0 conn(0x7faa8c0afc40 legacy=0x7faa8c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 1 monclient: found mon.noname-b 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0aba30 con 0x7faa8c06d8c0 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:58.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa84001a10 con 0x7faa8c06d8c0 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa89ffb640 10 monclient: dump: 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:58.251 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa8bfff640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.243+0000 7faa8bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.244+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa84001b70 con 0x7faa8c06d8c0 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.244+0000 7faa89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.244+0000 7faa8bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.244+0000 7faa8bfff640 4 set_mon_vals no callback set 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.245+0000 7faa89ffb640 1 -- 172.21.15.22:0/3433988092 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa84001fc0 con 0x7faa8c06d8c0 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.245+0000 7faa89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.245+0000 7faa89ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.245+0000 7faa89ffb640 10 monclient: dump: 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:58.252 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.245+0000 7faa8bfff640 10 monclient: shutdown 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.245+0000 7faa8bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:58.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.245+0000 7faa8bfff640 1 -- 172.21.15.22:0/3433988092 >> v1:172.21.15.22:6790/0 conn(0x7faa8c06d8c0 legacy=0x7faa8c0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.247+0000 7faa8bfff640 1 -- 172.21.15.22:0/3433988092 shutdown_connections 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.247+0000 7faa8bfff640 1 -- 172.21.15.22:0/3433988092 >> 172.21.15.22:0/3433988092 conn(0x7faa8c0673e0 msgr2=0x7faa8c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.247+0000 7faa8bfff640 1 -- 172.21.15.22:0/3433988092 shutdown_connections 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.248+0000 7faa8bfff640 1 -- 172.21.15.22:0/3433988092 wait complete. 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.249+0000 7faa8bfff640 10 monclient: build_initial_monmap 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.249+0000 7faa8bfff640 10 monclient: monmap: 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:44:58.256 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.249+0000 7faa8bfff640 1 librados: starting msgr at 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.249+0000 7faa8bfff640 1 librados: starting objecter 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.250+0000 7faa8bfff640 1 Processor -- start 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.250+0000 7faa8bfff640 1 -- start start 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.250+0000 7faa8bfff640 1 librados: setting wanted keys 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.250+0000 7faa8bfff640 1 librados: calling monclient init 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.250+0000 7faa8bfff640 10 monclient: init 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c06d8c0 addr v1:172.21.15.22:6789/0 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 10 monclient(hunting): picked mon.noname-c con 0x7faa8c0aad20 addr v1:172.21.15.22:6790/0 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c0afc40 addr v1:172.21.15.204:6789/0 2024-03-29T11:44:58.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c138cf0 con 0x7faa8c06d8c0 2024-03-29T11:44:58.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c13e760 con 0x7faa8c0afc40 2024-03-29T11:44:58.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c13d910 con 0x7faa8c0aad20 2024-03-29T11:44:58.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:58.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.251+0000 7faa8bfff640 10 monclient(hunting): authenticate will time out at 1548.527191s 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.253+0000 7faa8a7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7faa8c0aad20 0x7faa8c13ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57862/0 (socket says 172.21.15.22:57862) 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.253+0000 7faa8a7fc640 1 -- 172.21.15.22:0/3392771308 learned_addr learned my addr 172.21.15.22:0/3392771308 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 10 client.?.objecter ms_handle_connect 0x7faa8c0aad20 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 10 client.?.objecter ms_handle_connect 0x7faa8c0afc40 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625658418 0 0) 0x7faa8c13d910 con 0x7faa8c0aad20 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.254+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa78004eb0 con 0x7faa8c0aad20 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770254799 0 0) 0x7faa8c13e760 con 0x7faa8c0afc40 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c13d910 con 0x7faa8c0afc40 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879269869 0 0) 0x7faa8c13d910 con 0x7faa8c0afc40 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa8c13e760 con 0x7faa8c0afc40 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa7c001710 con 0x7faa8c0afc40 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 10 monclient(hunting): dump: 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.255+0000 7faa737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4141028818 0 0) 0x7faa8c13e760 con 0x7faa8c0afc40 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 >> v1:172.21.15.22:6790/0 conn(0x7faa8c0aad20 legacy=0x7faa8c13ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 >> v1:172.21.15.22:6789/0 conn(0x7faa8c06d8c0 legacy=0x7faa8c13d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 1 monclient: found mon.b 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c13fc80 con 0x7faa8c0afc40 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa7c001870 con 0x7faa8c0afc40 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7faa7c003790 con 0x7faa8c0afc40 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa737fe640 10 monclient: dump: 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.256+0000 7faa897fa640 4 set_mon_vals no callback set 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 10 monclient: _renew_subs 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c138860 con 0x7faa8c0afc40 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 10 monclient: _renew_subs 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c139090 con 0x7faa8c0afc40 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.257+0000 7faa8bfff640 1 librados: init done 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.259+0000 7faa8bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.259+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7faa7c019760 con 0x7faa8c0afc40 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.259+0000 7faa8bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.259+0000 7faa8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.260+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c13cf20 con 0x7faa8c0afc40 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.260+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1368017700 0 0) 0x7faa7c0836c0 con 0x7faa8c0afc40 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.260+0000 7faa737fe640 10 client.4546.objecter ms_dispatch 0x7faa8c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.260+0000 7faa737fe640 3 client.4546.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.260+0000 7faa737fe640 3 client.4546.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.260+0000 7faa737fe640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.260+0000 7faa8affd640 10 monclient: get_auth_request con 0x7faa7806dba0 auth_method 0 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.260+0000 7faa737fe640 10 client.4546.objecter ms_handle_connect 0x7faa7806dba0 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.266+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa7c04efa0 con 0x7faa8c0afc40 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.266+0000 7faa737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.266+0000 7faa737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:58.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.626+0000 7faa8bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:58.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.626+0000 7faa8bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:58.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.626+0000 7faa8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-29T11:44:58.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.626+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa8c1389f0 con 0x7faa8c0afc40 2024-03-29T11:44:58.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.628+0000 7faa737fe640 1 -- 172.21.15.22:0/3392771308 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1933163311) 0x7faa7c050ed0 con 0x7faa8c0afc40 2024-03-29T11:44:58.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.628+0000 7faa737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:58.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.628+0000 7faa737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:58.633 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:44:58.633 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:44:34.925774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:34.925774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:58.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.632+0000 7faa8bfff640 10 librados: watch_flush enter 2024-03-29T11:44:58.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.632+0000 7faa8bfff640 10 librados: watch_flush exit 2024-03-29T11:44:58.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.632+0000 7faa8bfff640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:44:58.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.632+0000 7faa8bfff640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:44:58.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.632+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 >> v1:172.21.15.204:6816/3848099522 conn(0x7faa7806dba0 legacy=0x7faa78070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.632+0000 7faa8bfff640 10 monclient: shutdown 2024-03-29T11:44:58.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.632+0000 7faa8bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:58.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.632+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 >> v1:172.21.15.204:6789/0 conn(0x7faa8c0afc40 legacy=0x7faa8c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.634+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 shutdown_connections 2024-03-29T11:44:58.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.634+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 >> 172.21.15.22:0/3392771308 conn(0x7faa8c0673e0 msgr2=0x7faa8c0a9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:58.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.636+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 shutdown_connections 2024-03-29T11:44:58.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.636+0000 7faa8bfff640 1 -- 172.21.15.22:0/3392771308 wait complete. 2024-03-29T11:44:58.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:44:58.636+0000 7faa8bfff640 1 librados: shutdown 2024-03-29T11:44:58.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:05.463 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:05.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.601+0000 7f9234a17640 10 monclient: get_monmap_and_config 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.601+0000 7f9234a17640 10 monclient: build_initial_monmap 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.601+0000 7f9234a17640 10 monclient: monmap: 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:45:05.601619+0000 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:45:05.601619+0000 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:05.605 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:45:05.606 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:45:05.606 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:45:05.606 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:05.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.602+0000 7f9234a17640 1 Processor -- start 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.602+0000 7f9234a17640 1 -- start start 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.602+0000 7f9234a17640 10 monclient: init 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.603+0000 7f9234a17640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.603+0000 7f9234a17640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.603+0000 7f9234a17640 10 monclient(hunting): picked mon.noname-c con 0x7f92300708a0 addr v1:172.21.15.204:6789/0 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.603+0000 7f9234a17640 10 monclient(hunting): picked mon.noname-a con 0x7f923006fe90 addr v1:172.21.15.22:6789/0 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.603+0000 7f922ed9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f9230143c50 0x7f923014b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49850/0 (socket says 172.21.15.22:49850) 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.604+0000 7f922ed9d640 1 -- 172.21.15.22:0/987734840 learned_addr learned my addr 172.21.15.22:0/987734840 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.604+0000 7f9234a17640 10 monclient(hunting): picked mon.noname-b con 0x7f9230143c50 addr v1:172.21.15.22:6790/0 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.604+0000 7f9234a17640 1 -- 172.21.15.22:0/987734840 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9230072630 con 0x7f923006fe90 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.604+0000 7f9234a17640 1 -- 172.21.15.22:0/987734840 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9230071410 con 0x7f92300708a0 2024-03-29T11:45:05.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.604+0000 7f9234a17640 1 -- 172.21.15.22:0/987734840 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92300afaa0 con 0x7f9230143c50 2024-03-29T11:45:05.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.604+0000 7f9234a17640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:05.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.604+0000 7f9234a17640 10 monclient(hunting): authenticate will time out at 1555.880477s 2024-03-29T11:45:05.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787962807 0 0) 0x7f92300afaa0 con 0x7f9230143c50 2024-03-29T11:45:05.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:05.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:05.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:05.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:45:05.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9218003520 con 0x7f9230143c50 2024-03-29T11:45:05.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147356968 0 0) 0x7f9218003520 con 0x7f9230143c50 2024-03-29T11:45:05.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 >> v1:172.21.15.204:6789/0 conn(0x7f92300708a0 legacy=0x7f92300af390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:05.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.606+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 >> v1:172.21.15.22:6789/0 conn(0x7f923006fe90 legacy=0x7f9230143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:05.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:45:05.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 1 monclient: found mon.noname-b 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92300ac8b0 con 0x7f9230143c50 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 10 monclient: _finish_auth 0 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9228001a10 con 0x7f9230143c50 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f9234a17640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f9234a17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f922d59a640 10 monclient: dump: 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:05.611 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.607+0000 7f9234a17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:05.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.608+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9228001be0 con 0x7f9230143c50 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.608+0000 7f922d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.608+0000 7f922d59a640 1 -- 172.21.15.22:0/987734840 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9228002030 con 0x7f9230143c50 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.608+0000 7f922d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.608+0000 7f922d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.608+0000 7f922d59a640 10 monclient: dump: 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:05.613 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:05.614 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:05.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.608+0000 7f9234a17640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:05.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.608+0000 7f9234a17640 4 set_mon_vals no callback set 2024-03-29T11:45:05.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.609+0000 7f9234a17640 10 monclient: shutdown 2024-03-29T11:45:05.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.609+0000 7f9234a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:05.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.609+0000 7f9234a17640 1 -- 172.21.15.22:0/987734840 >> v1:172.21.15.22:6790/0 conn(0x7f9230143c50 legacy=0x7f923014b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:05.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.610+0000 7f9234a17640 1 -- 172.21.15.22:0/987734840 shutdown_connections 2024-03-29T11:45:05.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.610+0000 7f9234a17640 1 -- 172.21.15.22:0/987734840 >> 172.21.15.22:0/987734840 conn(0x7f92300675b0 msgr2=0x7f92300b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:05.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.611+0000 7f9234a17640 1 -- 172.21.15.22:0/987734840 shutdown_connections 2024-03-29T11:45:05.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.611+0000 7f9234a17640 1 -- 172.21.15.22:0/987734840 wait complete. 2024-03-29T11:45:05.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.612+0000 7f9234a17640 10 monclient: build_initial_monmap 2024-03-29T11:45:05.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.612+0000 7f9234a17640 10 monclient: monmap: 2024-03-29T11:45:05.616 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:45:05.616 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:05.616 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:45:05.616 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:45:05.616 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:05.617 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:05.617 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:45:05.617 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:45:05.617 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:45:05.617 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:05.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.612+0000 7f9234a17640 1 librados: starting msgr at 2024-03-29T11:45:05.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.612+0000 7f9234a17640 1 librados: starting objecter 2024-03-29T11:45:05.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.612+0000 7f9234a17640 1 Processor -- start 2024-03-29T11:45:05.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.613+0000 7f9234a17640 1 -- start start 2024-03-29T11:45:05.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.613+0000 7f9234a17640 1 librados: setting wanted keys 2024-03-29T11:45:05.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.613+0000 7f9234a17640 1 librados: calling monclient init 2024-03-29T11:45:05.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.613+0000 7f9234a17640 10 monclient: init 2024-03-29T11:45:05.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:05.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:05.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 10 monclient(hunting): picked mon.noname-b con 0x7f923006fe90 addr v1:172.21.15.204:6789/0 2024-03-29T11:45:05.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 10 monclient(hunting): picked mon.noname-a con 0x7f92300708a0 addr v1:172.21.15.22:6789/0 2024-03-29T11:45:05.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 10 monclient(hunting): picked mon.noname-c con 0x7f9230143c50 addr v1:172.21.15.22:6790/0 2024-03-29T11:45:05.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92300aa8c0 con 0x7f92300708a0 2024-03-29T11:45:05.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92300a9520 con 0x7f923006fe90 2024-03-29T11:45:05.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92300a96b0 con 0x7f9230143c50 2024-03-29T11:45:05.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:05.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.615+0000 7f9234a17640 10 monclient(hunting): authenticate will time out at 1555.891164s 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.616+0000 7f922ed9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f9230143c50 0x7f92301df4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53908/0 (socket says 172.21.15.22:53908) 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.616+0000 7f922ed9d640 1 -- 172.21.15.22:0/1095751280 learned_addr learned my addr 172.21.15.22:0/1095751280 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.616+0000 7f921effd640 10 client.?.objecter ms_handle_connect 0x7f9230143c50 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.616+0000 7f921effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.617+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190330389 0 0) 0x7f92300a96b0 con 0x7f9230143c50 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.617+0000 7f921effd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.617+0000 7f921effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.617+0000 7f921effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.617+0000 7f921effd640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:45:05.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.617+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9210004eb0 con 0x7f9230143c50 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.617+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2600087623 0 0) 0x7f9210004eb0 con 0x7f9230143c50 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.618+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92300a96b0 con 0x7f9230143c50 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.618+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9228001ef0 con 0x7f9230143c50 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.618+0000 7f921effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.618+0000 7f921effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.618+0000 7f921effd640 10 monclient(hunting): dump: 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:05.623 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:05.624 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:05.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.618+0000 7f921effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:05.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.618+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1454762329 0 0) 0x7f92300a96b0 con 0x7f9230143c50 2024-03-29T11:45:05.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.618+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 >> v1:172.21.15.204:6789/0 conn(0x7f923006fe90 legacy=0x7f923013c430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:05.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 >> v1:172.21.15.22:6789/0 conn(0x7f92300708a0 legacy=0x7f92301dedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 1 monclient: found mon.c 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923013cb40 con 0x7f9230143c50 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 10 monclient: _finish_auth 0 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f921effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 10 monclient: _renew_subs 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923013cd90 con 0x7f9230143c50 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:05.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 10 monclient: _renew_subs 2024-03-29T11:45:05.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:05.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92300a9b50 con 0x7f9230143c50 2024-03-29T11:45:05.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.620+0000 7f9234a17640 1 librados: init done 2024-03-29T11:45:05.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.621+0000 7f921effd640 10 client.4670.objecter ms_handle_connect 0x7f92300708a0 2024-03-29T11:45:05.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.621+0000 7f921effd640 10 client.4670.objecter resend_mon_ops 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.621+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9228004f70 con 0x7f9230143c50 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.621+0000 7f921effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.621+0000 7f921ffff640 4 set_mon_vals no callback set 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.622+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f9228005280 con 0x7f9230143c50 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.622+0000 7f921effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.622+0000 7f921effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.622+0000 7f921effd640 10 monclient: dump: 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:05.627 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.623+0000 7f9234a17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.623+0000 7f9234a17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.623+0000 7f9234a17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:05.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.623+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9214005080 con 0x7f9230143c50 2024-03-29T11:45:05.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.624+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f922801b250 con 0x7f9230143c50 2024-03-29T11:45:05.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.624+0000 7f922e59c640 10 monclient: get_auth_request con 0x7f921006db30 auth_method 0 2024-03-29T11:45:05.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.624+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1368017700 0 0) 0x7f9228084f80 con 0x7f9230143c50 2024-03-29T11:45:05.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.624+0000 7f921effd640 10 client.4670.objecter ms_dispatch 0x7f9230000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:05.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.624+0000 7f921effd640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:05.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.624+0000 7f921effd640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:05.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.625+0000 7f921effd640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-29T11:45:05.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.625+0000 7f921effd640 10 client.4670.objecter ms_handle_connect 0x7f921006db30 2024-03-29T11:45:05.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.635+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9228050860 con 0x7f9230143c50 2024-03-29T11:45:05.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.635+0000 7f921effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:05.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.635+0000 7f921effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:05.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.991+0000 7f9234a17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:05.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.991+0000 7f9234a17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:05.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.991+0000 7f9234a17640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:05.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.991+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9214005330 con 0x7f9230143c50 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.994+0000 7f921effd640 1 -- 172.21.15.22:0/1095751280 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1933163311) 0x7f9228052790 con 0x7f9230143c50 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.994+0000 7f921effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:05.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.994+0000 7f921effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:05.998 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:45:05.998 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:44:34.925774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:34.925774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 10 librados: watch_flush enter 2024-03-29T11:45:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 10 librados: watch_flush exit 2024-03-29T11:45:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-29T11:45:06.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 10 client.4670.objecter successfully canceled tick 2024-03-29T11:45:06.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 >> v1:172.21.15.204:6816/3848099522 conn(0x7f921006db30 legacy=0x7f921006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:06.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 10 monclient: shutdown 2024-03-29T11:45:06.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:06.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 >> v1:172.21.15.22:6790/0 conn(0x7f9230143c50 legacy=0x7f92301df4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:06.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 shutdown_connections 2024-03-29T11:45:06.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.998+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 >> 172.21.15.22:0/1095751280 conn(0x7f92300675b0 msgr2=0x7f92300af300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:05.999+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 shutdown_connections 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:06.000+0000 7f9234a17640 1 -- 172.21.15.22:0/1095751280 wait complete. 2024-03-29T11:45:06.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:06.000+0000 7f9234a17640 1 librados: shutdown 2024-03-29T11:45:06.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:12.829 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:12.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.969+0000 7ff23dcb3640 10 monclient: get_monmap_and_config 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.969+0000 7ff23dcb3640 10 monclient: build_initial_monmap 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.969+0000 7ff23dcb3640 10 monclient: monmap: 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:45:12.970395+0000 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:45:12.970395+0000 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:45:12.974 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:12.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.970+0000 7ff23dcb3640 1 Processor -- start 2024-03-29T11:45:12.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.971+0000 7ff23dcb3640 1 -- start start 2024-03-29T11:45:12.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.971+0000 7ff23dcb3640 10 monclient: init 2024-03-29T11:45:12.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:12.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:12.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 10 monclient(hunting): picked mon.noname-a con 0x7ff2380b0860 addr v1:172.21.15.22:6789/0 2024-03-29T11:45:12.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 10 monclient(hunting): picked mon.noname-c con 0x7ff238072450 addr v1:172.21.15.204:6789/0 2024-03-29T11:45:12.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 10 monclient(hunting): picked mon.noname-b con 0x7ff23813ee40 addr v1:172.21.15.22:6790/0 2024-03-29T11:45:12.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2380ad840 con 0x7ff2380b0860 2024-03-29T11:45:12.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff2377fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff2380b0860 0x7ff2380acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:40096/0 (socket says 172.21.15.22:40096) 2024-03-29T11:45:12.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff2377fe640 1 -- 172.21.15.22:0/2811442316 learned_addr learned my addr 172.21.15.22:0/2811442316 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:45:12.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 1 -- 172.21.15.22:0/2811442316 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23813c830 con 0x7ff238072450 2024-03-29T11:45:12.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 1 -- 172.21.15.22:0/2811442316 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23813c9c0 con 0x7ff23813ee40 2024-03-29T11:45:12.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:12.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff23dcb3640 10 monclient(hunting): authenticate will time out at 1563.249606s 2024-03-29T11:45:12.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329734356 0 0) 0x7ff2380ad840 con 0x7ff2380b0860 2024-03-29T11:45:12.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff2367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:12.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff2367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:12.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff2367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff2367fc640 10 monclient(hunting): my global_id is 4611 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.972+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff220003520 con 0x7ff2380b0860 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 914704918 0 0) 0x7ff220003520 con 0x7ff2380b0860 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 >> v1:172.21.15.22:6790/0 conn(0x7ff23813ee40 legacy=0x7ff23813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 >> v1:172.21.15.204:6789/0 conn(0x7ff238072450 legacy=0x7ff23813c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 1 monclient: found mon.noname-a 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23813f970 con 0x7ff2380b0860 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:12.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff230001a10 con 0x7ff2380b0860 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: dump: 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:12.983 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff23dcb3640 5 monclient: authenticate success, global_id 4611 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff23dcb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff230001b70 con 0x7ff2380b0860 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 1 -- 172.21.15.22:0/2811442316 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff230001f80 con 0x7ff2380b0860 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff23dcb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff2367fc640 10 monclient: dump: 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.984 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.973+0000 7ff23dcb3640 4 set_mon_vals no callback set 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.974+0000 7ff23dcb3640 10 monclient: shutdown 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.974+0000 7ff23dcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.974+0000 7ff23dcb3640 1 -- 172.21.15.22:0/2811442316 >> v1:172.21.15.22:6789/0 conn(0x7ff2380b0860 legacy=0x7ff2380acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.974+0000 7ff23dcb3640 1 -- 172.21.15.22:0/2811442316 shutdown_connections 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.974+0000 7ff23dcb3640 1 -- 172.21.15.22:0/2811442316 >> 172.21.15.22:0/2811442316 conn(0x7ff23806b660 msgr2=0x7ff23806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.974+0000 7ff23dcb3640 1 -- 172.21.15.22:0/2811442316 shutdown_connections 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.974+0000 7ff23dcb3640 1 -- 172.21.15.22:0/2811442316 wait complete. 2024-03-29T11:45:12.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.975+0000 7ff23dcb3640 10 monclient: build_initial_monmap 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.975+0000 7ff23dcb3640 10 monclient: monmap: 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.976+0000 7ff23dcb3640 1 librados: starting msgr at 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.976+0000 7ff23dcb3640 1 librados: starting objecter 2024-03-29T11:45:12.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.976+0000 7ff23dcb3640 1 Processor -- start 2024-03-29T11:45:12.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.976+0000 7ff23dcb3640 1 -- start start 2024-03-29T11:45:12.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.977+0000 7ff23dcb3640 1 librados: setting wanted keys 2024-03-29T11:45:12.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.977+0000 7ff23dcb3640 1 librados: calling monclient init 2024-03-29T11:45:12.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.977+0000 7ff23dcb3640 10 monclient: init 2024-03-29T11:45:12.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:12.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:12.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 10 monclient(hunting): picked mon.noname-b con 0x7ff238072450 addr v1:172.21.15.204:6789/0 2024-03-29T11:45:12.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 10 monclient(hunting): picked mon.noname-c con 0x7ff2380b0860 addr v1:172.21.15.22:6790/0 2024-03-29T11:45:12.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 10 monclient(hunting): picked mon.noname-a con 0x7ff23813ee40 addr v1:172.21.15.22:6789/0 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff238134580 con 0x7ff23813ee40 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2380b2170 con 0x7ff238072450 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2380b15b0 con 0x7ff2380b0860 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.978+0000 7ff23dcb3640 10 monclient(hunting): authenticate will time out at 1563.255843s 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff236ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff2380b0860 0x7ff2381d4230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52874/0 (socket says 172.21.15.22:52874) 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff236ffd640 1 -- 172.21.15.22:0/935134457 learned_addr learned my addr 172.21.15.22:0/935134457 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff21ffff640 10 client.?.objecter ms_handle_connect 0x7ff23813ee40 2024-03-29T11:45:12.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff21ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff21ffff640 10 client.?.objecter ms_handle_connect 0x7ff238072450 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff21ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff21ffff640 10 client.?.objecter ms_handle_connect 0x7ff2380b0860 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff21ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.979+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597031460 0 0) 0x7ff238134580 con 0x7ff23813ee40 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff224004eb0 con 0x7ff23813ee40 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445561461 0 0) 0x7ff2380b15b0 con 0x7ff2380b0860 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:12.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff238134580 con 0x7ff2380b0860 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295867608 0 0) 0x7ff2380b2170 con 0x7ff238072450 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2380b15b0 con 0x7ff238072450 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1865667934 0 0) 0x7ff238134580 con 0x7ff2380b0860 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2380b2170 con 0x7ff2380b0860 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2054722528 0 0) 0x7ff224004eb0 con 0x7ff23813ee40 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff238134580 con 0x7ff23813ee40 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff2280018b0 con 0x7ff2380b0860 2024-03-29T11:45:12.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.980+0000 7ff21ffff640 10 monclient(hunting): dump: 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:12.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff22c001790 con 0x7ff23813ee40 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient(hunting): dump: 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1657143281 0 0) 0x7ff2380b2170 con 0x7ff2380b0860 2024-03-29T11:45:12.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 >> v1:172.21.15.204:6789/0 conn(0x7ff238072450 legacy=0x7ff2381da3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 >> v1:172.21.15.22:6789/0 conn(0x7ff23813ee40 legacy=0x7ff2380b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 1 monclient: found mon.c 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2381d4940 con 0x7ff2380b0860 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient: _finish_auth 0 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 5 monclient: authenticate success, global_id 4676 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 10 monclient: _renew_subs 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:12.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2380b11d0 con 0x7ff2380b0860 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff228003610 con 0x7ff2380b0860 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7ff2280039e0 con 0x7ff2380b0860 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff21ffff640 10 monclient: dump: 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:12.999 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff235ffb640 4 set_mon_vals no callback set 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 10 monclient: _renew_subs 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2380b4380 con 0x7ff2380b0860 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.981+0000 7ff23dcb3640 1 librados: init done 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.983+0000 7ff23dcb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.983+0000 7ff23dcb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.983+0000 7ff23dcb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.983+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2380b0c40 con 0x7ff2380b0860 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.983+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7ff228003b40 con 0x7ff2380b0860 2024-03-29T11:45:13.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.984+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1368017700 0 0) 0x7ff228084800 con 0x7ff2380b0860 2024-03-29T11:45:13.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.984+0000 7ff21ffff640 10 client.4676.objecter ms_dispatch 0x7ff238000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:13.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.984+0000 7ff21ffff640 3 client.4676.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:13.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.984+0000 7ff21ffff640 3 client.4676.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:13.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.984+0000 7ff21ffff640 20 client.4676.objecter dump_active .. 0 homeless 2024-03-29T11:45:13.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.984+0000 7ff2377fe640 10 monclient: get_auth_request con 0x7ff22406e220 auth_method 0 2024-03-29T11:45:13.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.985+0000 7ff21ffff640 10 client.4676.objecter ms_handle_connect 0x7ff22406e220 2024-03-29T11:45:13.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.991+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2280500e0 con 0x7ff2380b0860 2024-03-29T11:45:13.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.992+0000 7ff21ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:13.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:12.992+0000 7ff21ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:13.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.354+0000 7ff23dcb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.354+0000 7ff23dcb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.354+0000 7ff23dcb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:13.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.354+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff238075f10 con 0x7ff2380b0860 2024-03-29T11:45:13.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.355+0000 7ff21ffff640 1 -- 172.21.15.22:0/935134457 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1933163311) 0x7ff228052010 con 0x7ff2380b0860 2024-03-29T11:45:13.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.355+0000 7ff21ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.355+0000 7ff21ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:13.360 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:45:13.360 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:44:34.925774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:34.925774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 10 librados: watch_flush enter 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 10 librados: watch_flush exit 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 20 client.4676.objecter shutdown clearing up homeless session... 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 10 client.4676.objecter successfully canceled tick 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 >> v1:172.21.15.204:6816/3848099522 conn(0x7ff22406e220 legacy=0x7ff2240706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 10 monclient: shutdown 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 >> v1:172.21.15.22:6790/0 conn(0x7ff2380b0860 legacy=0x7ff2381d4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 shutdown_connections 2024-03-29T11:45:13.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 >> 172.21.15.22:0/935134457 conn(0x7ff23806b660 msgr2=0x7ff2380a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 shutdown_connections 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.359+0000 7ff23dcb3640 1 -- 172.21.15.22:0/935134457 wait complete. 2024-03-29T11:45:13.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:13.360+0000 7ff23dcb3640 1 librados: shutdown 2024-03-29T11:45:13.379 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:20.186 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:20.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.325+0000 7f1f95624640 10 monclient: get_monmap_and_config 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.325+0000 7f1f95624640 10 monclient: build_initial_monmap 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.325+0000 7f1f95624640 10 monclient: monmap: 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:45:20.326356+0000 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:45:20.326356+0000 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:45:20.331 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:45:20.332 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:45:20.332 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:20.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.326+0000 7f1f95624640 1 Processor -- start 2024-03-29T11:45:20.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.326+0000 7f1f95624640 1 -- start start 2024-03-29T11:45:20.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.326+0000 7f1f95624640 10 monclient: init 2024-03-29T11:45:20.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:20.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:20.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 10 monclient(hunting): picked mon.noname-c con 0x7f1f900a9750 addr v1:172.21.15.204:6789/0 2024-03-29T11:45:20.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 10 monclient(hunting): picked mon.noname-b con 0x7f1f900adcf0 addr v1:172.21.15.22:6790/0 2024-03-29T11:45:20.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 10 monclient(hunting): picked mon.noname-a con 0x7f1f900aad20 addr v1:172.21.15.22:6789/0 2024-03-29T11:45:20.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f90149870 con 0x7f1f900aad20 2024-03-29T11:45:20.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900abf80 con 0x7f1f900a9750 2024-03-29T11:45:20.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f90149ac0 con 0x7f1f900adcf0 2024-03-29T11:45:20.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:20.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.327+0000 7f1f95624640 10 monclient(hunting): authenticate will time out at 1570.604506s 2024-03-29T11:45:20.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.334+0000 7f1f94e23640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f1f900aad20 0x7f1f90151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:37159/0 (socket says 172.21.15.22:37159) 2024-03-29T11:45:20.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.334+0000 7f1f94e23640 1 -- 172.21.15.22:0/425920074 learned_addr learned my addr 172.21.15.22:0/425920074 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:45:20.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603776855 0 0) 0x7f1f90149870 con 0x7f1f900aad20 2024-03-29T11:45:20.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 10 monclient(hunting): my global_id is 4617 2024-03-29T11:45:20.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f78003520 con 0x7f1f900aad20 2024-03-29T11:45:20.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866315364 0 0) 0x7f1f900abf80 con 0x7f1f900a9750 2024-03-29T11:45:20.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:45:20.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.335+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f90149870 con 0x7f1f900a9750 2024-03-29T11:45:20.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2505926352 0 0) 0x7f1f78003520 con 0x7f1f900aad20 2024-03-29T11:45:20.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 >> v1:172.21.15.22:6790/0 conn(0x7f1f900adcf0 legacy=0x7f1f90143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 >> v1:172.21.15.204:6789/0 conn(0x7f1f900a9750 legacy=0x7f1f90149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:20.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 1 monclient: found mon.noname-a 2024-03-29T11:45:20.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:20.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:45:20.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9014a6f0 con 0x7f1f900aad20 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 10 monclient: _finish_auth 0 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1f88001a10 con 0x7f1f900aad20 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f8effd640 10 monclient: dump: 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:20.345 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:20.346 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:20.346 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:20.346 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:20.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f95624640 5 monclient: authenticate success, global_id 4617 2024-03-29T11:45:20.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.339+0000 7f1f95624640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:20.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.340+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f88001b70 con 0x7f1f900aad20 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.340+0000 7f1f8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.340+0000 7f1f95624640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.340+0000 7f1f95624640 4 set_mon_vals no callback set 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.340+0000 7f1f8effd640 1 -- 172.21.15.22:0/425920074 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1f88001fc0 con 0x7f1f900aad20 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.340+0000 7f1f8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.340+0000 7f1f8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.340+0000 7f1f8effd640 10 monclient: dump: 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:20.347 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:20.348 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:20.348 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:20.348 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:20.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.341+0000 7f1f95624640 10 monclient: shutdown 2024-03-29T11:45:20.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.341+0000 7f1f95624640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:20.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.341+0000 7f1f95624640 1 -- 172.21.15.22:0/425920074 >> v1:172.21.15.22:6789/0 conn(0x7f1f900aad20 legacy=0x7f1f90151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.343+0000 7f1f95624640 1 -- 172.21.15.22:0/425920074 shutdown_connections 2024-03-29T11:45:20.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.343+0000 7f1f95624640 1 -- 172.21.15.22:0/425920074 >> 172.21.15.22:0/425920074 conn(0x7f1f900673e0 msgr2=0x7f1f900b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:20.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.344+0000 7f1f95624640 1 -- 172.21.15.22:0/425920074 shutdown_connections 2024-03-29T11:45:20.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.344+0000 7f1f95624640 1 -- 172.21.15.22:0/425920074 wait complete. 2024-03-29T11:45:20.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.345+0000 7f1f95624640 10 monclient: build_initial_monmap 2024-03-29T11:45:20.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.345+0000 7f1f95624640 10 monclient: monmap: 2024-03-29T11:45:20.350 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.345+0000 7f1f95624640 1 librados: starting msgr at 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.345+0000 7f1f95624640 1 librados: starting objecter 2024-03-29T11:45:20.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.346+0000 7f1f95624640 1 Processor -- start 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.346+0000 7f1f95624640 1 -- start start 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.346+0000 7f1f95624640 1 librados: setting wanted keys 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.346+0000 7f1f95624640 1 librados: calling monclient init 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.346+0000 7f1f95624640 10 monclient: init 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 10 monclient(hunting): picked mon.noname-c con 0x7f1f900a9750 addr v1:172.21.15.22:6790/0 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 10 monclient(hunting): picked mon.noname-b con 0x7f1f900aad20 addr v1:172.21.15.204:6789/0 2024-03-29T11:45:20.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 10 monclient(hunting): picked mon.noname-a con 0x7f1f900adcf0 addr v1:172.21.15.22:6789/0 2024-03-29T11:45:20.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900b02a0 con 0x7f1f900adcf0 2024-03-29T11:45:20.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900b1140 con 0x7f1f900aad20 2024-03-29T11:45:20.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900afd80 con 0x7f1f900a9750 2024-03-29T11:45:20.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:20.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.347+0000 7f1f95624640 10 monclient(hunting): authenticate will time out at 1570.624409s 2024-03-29T11:45:20.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.351+0000 7f1f8f7fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f1f900aad20 0x7f1f901de5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.22:47421/0 (socket says 172.21.15.22:47421) 2024-03-29T11:45:20.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.351+0000 7f1f8f7fe640 1 -- 172.21.15.22:0/976815463 learned_addr learned my addr 172.21.15.22:0/976815463 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:45:20.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 client.?.objecter ms_handle_connect 0x7f1f900aad20 2024-03-29T11:45:20.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:20.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274272923 0 0) 0x7f1f900b1140 con 0x7f1f900aad20 2024-03-29T11:45:20.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f7c004eb0 con 0x7f1f900aad20 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 client.?.objecter ms_handle_connect 0x7f1f900adcf0 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704128779 0 0) 0x7f1f900b02a0 con 0x7f1f900adcf0 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 10 monclient(hunting): my global_id is 4620 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.352+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f900b1140 con 0x7f1f900adcf0 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 130852474 0 0) 0x7f1f900b1140 con 0x7f1f900adcf0 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f900b02a0 con 0x7f1f900adcf0 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1f88001e80 con 0x7f1f900adcf0 2024-03-29T11:45:20.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 10 monclient(hunting): dump: 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:20.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931799588 0 0) 0x7f1f7c004eb0 con 0x7f1f900aad20 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f900b1140 con 0x7f1f900aad20 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2992200504 0 0) 0x7f1f900b02a0 con 0x7f1f900adcf0 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 >> v1:172.21.15.22:6790/0 conn(0x7f1f900a9750 legacy=0x7f1f9013c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 >> v1:172.21.15.204:6789/0 conn(0x7f1f900aad20 legacy=0x7f1f901de5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 monclient: found mon.a 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f900af8a0 con 0x7f1f900adcf0 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 10 monclient: _finish_auth 0 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.353+0000 7f1f8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 5 monclient: authenticate success, global_id 4620 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 10 monclient: _renew_subs 2024-03-29T11:45:20.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f900afa90 con 0x7f1f900adcf0 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 10 client.4620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 10 monclient: _renew_subs 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9013ca60 con 0x7f1f900adcf0 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f95624640 1 librados: init done 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f88001520 con 0x7f1f900adcf0 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7f1f88004f70 con 0x7f1f900adcf0 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f8cff9640 10 monclient: dump: 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:20.364 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.354+0000 7f1f8e7fc640 4 set_mon_vals no callback set 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.355+0000 7f1f95624640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.355+0000 7f1f95624640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.355+0000 7f1f95624640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:45:20.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.355+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f70005080 con 0x7f1f900adcf0 2024-03-29T11:45:20.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.356+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7f1f8801af40 con 0x7f1f900adcf0 2024-03-29T11:45:20.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.356+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1368017700 0 0) 0x7f1f88084c70 con 0x7f1f900adcf0 2024-03-29T11:45:20.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.356+0000 7f1f8cff9640 10 client.4620.objecter ms_dispatch 0x7f1f90000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:20.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.356+0000 7f1f8cff9640 3 client.4620.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:20.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.356+0000 7f1f8cff9640 3 client.4620.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:20.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.357+0000 7f1f8cff9640 20 client.4620.objecter dump_active .. 0 homeless 2024-03-29T11:45:20.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.358+0000 7f1f8ffff640 10 monclient: get_auth_request con 0x7f1f7c06de10 auth_method 0 2024-03-29T11:45:20.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.359+0000 7f1f8cff9640 10 client.4620.objecter ms_handle_connect 0x7f1f7c06de10 2024-03-29T11:45:20.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.367+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f88050550 con 0x7f1f900adcf0 2024-03-29T11:45:20.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.367+0000 7f1f8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.367+0000 7f1f8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:20.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.724+0000 7f1f95624640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:20.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.724+0000 7f1f95624640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:20.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.724+0000 7f1f95624640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:45:20.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.724+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f70005330 con 0x7f1f900adcf0 2024-03-29T11:45:20.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.725+0000 7f1f8cff9640 1 -- 172.21.15.22:0/976815463 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1933163311) 0x7f1f88052480 con 0x7f1f900adcf0 2024-03-29T11:45:20.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.725+0000 7f1f8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:20.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.725+0000 7f1f8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:20.730 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:45:20.730 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:44:34.925774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:34.925774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:20.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.728+0000 7f1f95624640 10 librados: watch_flush enter 2024-03-29T11:45:20.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 10 librados: watch_flush exit 2024-03-29T11:45:20.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 20 client.4620.objecter shutdown clearing up homeless session... 2024-03-29T11:45:20.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 10 client.4620.objecter successfully canceled tick 2024-03-29T11:45:20.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 >> v1:172.21.15.204:6816/3848099522 conn(0x7f1f7c06de10 legacy=0x7f1f7c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 10 monclient: shutdown 2024-03-29T11:45:20.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:20.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 >> v1:172.21.15.22:6789/0 conn(0x7f1f900adcf0 legacy=0x7f1f901e6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 shutdown_connections 2024-03-29T11:45:20.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.729+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 >> 172.21.15.22:0/976815463 conn(0x7f1f900673e0 msgr2=0x7f1f90069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:20.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.730+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 shutdown_connections 2024-03-29T11:45:20.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.730+0000 7f1f95624640 1 -- 172.21.15.22:0/976815463 wait complete. 2024-03-29T11:45:20.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:20.730+0000 7f1f95624640 1 librados: shutdown 2024-03-29T11:45:20.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:27.557 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.707+0000 7febcda4a640 10 monclient: get_monmap_and_config 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.707+0000 7febcda4a640 10 monclient: build_initial_monmap 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.707+0000 7febcda4a640 10 monclient: monmap: 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:45:27.708906+0000 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:45:27.708906+0000 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:45:27.713 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-29T11:45:27.714 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:45:27.714 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:27.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.709+0000 7febcda4a640 1 Processor -- start 2024-03-29T11:45:27.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.709+0000 7febcda4a640 1 -- start start 2024-03-29T11:45:27.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.709+0000 7febcda4a640 10 monclient: init 2024-03-29T11:45:27.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.709+0000 7febcda4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:27.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.709+0000 7febcda4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:27.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.710+0000 7febcda4a640 10 monclient(hunting): picked mon.noname-b con 0x7febc81457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:45:27.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febc6ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7febc81457f0 0x7febc813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56360/0 (socket says 172.21.15.22:56360) 2024-03-29T11:45:27.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febcda4a640 10 monclient(hunting): picked mon.noname-a con 0x7febc813f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:45:27.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febc6ffd640 1 -- 172.21.15.22:0/1458479335 learned_addr learned my addr 172.21.15.22:0/1458479335 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:45:27.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febcda4a640 10 monclient(hunting): picked mon.noname-c con 0x7febc80a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:45:27.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febcda4a640 1 -- 172.21.15.22:0/1458479335 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc80aaef0 con 0x7febc813f650 2024-03-29T11:45:27.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febcda4a640 1 -- 172.21.15.22:0/1458479335 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8145bd0 con 0x7febc80a9d70 2024-03-29T11:45:27.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febcda4a640 1 -- 172.21.15.22:0/1458479335 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8145eb0 con 0x7febc81457f0 2024-03-29T11:45:27.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febcda4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:27.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.712+0000 7febcda4a640 10 monclient(hunting): authenticate will time out at 1577.989994s 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.713+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266693359 0 0) 0x7febc8145eb0 con 0x7febc81457f0 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.713+0000 7febc67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.713+0000 7febc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.713+0000 7febc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.713+0000 7febc67fc640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.713+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb4003520 con 0x7febc81457f0 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.713+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381081166 0 0) 0x7febc80aaef0 con 0x7febc813f650 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.714+0000 7febc67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.714+0000 7febc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.714+0000 7febc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:27.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.714+0000 7febc67fc640 10 monclient(hunting): my global_id is 4623 2024-03-29T11:45:27.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.714+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc8145eb0 con 0x7febc813f650 2024-03-29T11:45:27.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.714+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 269478728 0 0) 0x7febc8145eb0 con 0x7febc813f650 2024-03-29T11:45:27.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.714+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 >> v1:172.21.15.22:6790/0 conn(0x7febc81457f0 legacy=0x7febc813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:27.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.714+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 >> v1:172.21.15.204:6789/0 conn(0x7febc80a9d70 legacy=0x7febc814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:27.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:27.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 1 monclient: found mon.noname-a 2024-03-29T11:45:27.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:27.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8146340 con 0x7febc813f650 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7febb8001670 con 0x7febc813f650 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: dump: 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:27.722 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febcda4a640 5 monclient: authenticate success, global_id 4623 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febcda4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febb80017d0 con 0x7febc813f650 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 1 -- 172.21.15.22:0/1458479335 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7febb8001c20 con 0x7febc813f650 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febc67fc640 10 monclient: dump: 2024-03-29T11:45:27.723 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febcda4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:27.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.716+0000 7febcda4a640 4 set_mon_vals no callback set 2024-03-29T11:45:27.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.717+0000 7febcda4a640 10 monclient: shutdown 2024-03-29T11:45:27.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.717+0000 7febcda4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:27.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.717+0000 7febcda4a640 1 -- 172.21.15.22:0/1458479335 >> v1:172.21.15.22:6789/0 conn(0x7febc813f650 legacy=0x7febc80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:27.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.719+0000 7febcda4a640 1 -- 172.21.15.22:0/1458479335 shutdown_connections 2024-03-29T11:45:27.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.719+0000 7febcda4a640 1 -- 172.21.15.22:0/1458479335 >> 172.21.15.22:0/1458479335 conn(0x7febc8066a00 msgr2=0x7febc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:27.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.720+0000 7febcda4a640 1 -- 172.21.15.22:0/1458479335 shutdown_connections 2024-03-29T11:45:27.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.720+0000 7febcda4a640 1 -- 172.21.15.22:0/1458479335 wait complete. 2024-03-29T11:45:27.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.721+0000 7febcda4a640 10 monclient: build_initial_monmap 2024-03-29T11:45:27.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.721+0000 7febcda4a640 10 monclient: monmap: 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.722+0000 7febcda4a640 1 librados: starting msgr at 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.722+0000 7febcda4a640 1 librados: starting objecter 2024-03-29T11:45:27.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.722+0000 7febcda4a640 1 Processor -- start 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.723+0000 7febcda4a640 1 -- start start 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.723+0000 7febcda4a640 1 librados: setting wanted keys 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.723+0000 7febcda4a640 1 librados: calling monclient init 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.723+0000 7febcda4a640 10 monclient: init 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 10 monclient(hunting): picked mon.noname-b con 0x7febc80a9d70 addr v1:172.21.15.204:6789/0 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 10 monclient(hunting): picked mon.noname-a con 0x7febc813f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:45:27.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 10 monclient(hunting): picked mon.noname-c con 0x7febc81457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8139f90 con 0x7febc813f650 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8141fa0 con 0x7febc80a9d70 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8142280 con 0x7febc81457f0 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.724+0000 7febcda4a640 10 monclient(hunting): authenticate will time out at 1578.001518s 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc77fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7febc81457f0 0x7febc81e6bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58008/0 (socket says 172.21.15.22:58008) 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc77fe640 1 -- 172.21.15.22:0/2751827598 learned_addr learned my addr 172.21.15.22:0/2751827598 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc4ff9640 10 client.?.objecter ms_handle_connect 0x7febc81457f0 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30324610 0 0) 0x7febc8142280 con 0x7febc81457f0 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:27.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc4ff9640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.725+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba8004eb0 con 0x7febc81457f0 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2867389346 0 0) 0x7feba8004eb0 con 0x7febc81457f0 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc8142280 con 0x7febc81457f0 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7febc0001b70 con 0x7febc81457f0 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 10 monclient(hunting): dump: 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 522691269 0 0) 0x7febc8142280 con 0x7febc81457f0 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.726+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 >> v1:172.21.15.204:6789/0 conn(0x7febc80a9d70 legacy=0x7febc813fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 >> v1:172.21.15.22:6789/0 conn(0x7febc813f650 legacy=0x7febc81e34a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 1 monclient: found mon.c 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8072040 con 0x7febc81457f0 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 10 monclient: _renew_subs 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc813b810 con 0x7febc81457f0 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 10 monclient: _renew_subs 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febc8142bf0 con 0x7febc81457f0 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.729+0000 7febcda4a640 1 librados: init done 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.730+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febc0001cd0 con 0x7febc81457f0 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.730+0000 7febc4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.730+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2497171689 0 0) 0x7febc0004e20 con 0x7febc81457f0 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.730+0000 7febc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.730+0000 7febc4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.730+0000 7febc4ff9640 10 monclient: dump: 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:fsid 5b52d532-7394-417c-b834-972932886c7f 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:33:53.819653+0000 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.730+0000 7febc57fa640 4 set_mon_vals no callback set 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.731+0000 7febcda4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.731+0000 7febcda4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.731+0000 7febcda4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.731+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac005080 con 0x7febc81457f0 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.732+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 18911957 0 0) 0x7febc0004f80 con 0x7febc81457f0 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.733+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1368017700 0 0) 0x7febc0084d90 con 0x7febc81457f0 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.733+0000 7febc4ff9640 10 client.4685.objecter ms_dispatch 0x7febc8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.733+0000 7febc4ff9640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.733+0000 7febc4ff9640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.733+0000 7febc4ff9640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.733+0000 7febc6ffd640 10 monclient: get_auth_request con 0x7feba806dd20 auth_method 0 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.734+0000 7febc4ff9640 10 client.4685.objecter ms_handle_connect 0x7feba806dd20 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.741+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febc0050670 con 0x7febc81457f0 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.741+0000 7febc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:27.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:27.741+0000 7febc4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:28.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.081+0000 7febcda4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.081+0000 7febcda4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.081+0000 7febcda4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:45:28.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.081+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febac005330 con 0x7febc81457f0 2024-03-29T11:45:28.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.082+0000 7febc4ff9640 1 -- 172.21.15.22:0/2751827598 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1933163311) 0x7febc00525a0 con 0x7febc81457f0 2024-03-29T11:45:28.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.082+0000 7febc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.082+0000 7febc4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:28.089 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:45:28.089 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"5b52d532-7394-417c-b834-972932886c7f","created":"2024-03-29T11:34:29.988387+0000","modified":"2024-03-29T11:44:34.925774+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:44:34.925774+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8a925ac5-9ea5-4bd1-84cf-9415000e9422","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6bcbc966-5d5a-4edc-954f-b404f3090523","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"99bf3692-4e61-4936-b8ec-082a3bd7b4d4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8de7905a-e6f7-43ab-896e-4c9dddb96521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b730bd7f-7744-45f5-979d-4fbc1904ad26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"91c34317-366b-402e-8cab-d9e594d0273b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c9df31e9-6a29-42fb-8314-2e7d0a11ced5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f54e98c9-12b9-4c5b-8939-a54cafe12e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:28.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.086+0000 7febcda4a640 10 librados: watch_flush enter 2024-03-29T11:45:28.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.086+0000 7febcda4a640 10 librados: watch_flush exit 2024-03-29T11:45:28.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.086+0000 7febcda4a640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-29T11:45:28.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.086+0000 7febcda4a640 10 client.4685.objecter successfully canceled tick 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.086+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 >> v1:172.21.15.204:6816/3848099522 conn(0x7feba806dd20 legacy=0x7feba80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.086+0000 7febcda4a640 10 monclient: shutdown 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.086+0000 7febcda4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.086+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 >> v1:172.21.15.22:6790/0 conn(0x7febc81457f0 legacy=0x7febc81e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.087+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 shutdown_connections 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.087+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 >> 172.21.15.22:0/2751827598 conn(0x7febc8066a00 msgr2=0x7febc80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.089+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 shutdown_connections 2024-03-29T11:45:28.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.091+0000 7febcda4a640 1 -- 172.21.15.22:0/2751827598 wait complete. 2024-03-29T11:45:28.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:45:28.093+0000 7febcda4a640 1 librados: shutdown 2024-03-29T11:45:28.115 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:28.115 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:45:28.117 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:45:28.117 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:45:28.117 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.152 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:45:28.152 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:45:28.152 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.226 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:45:28.227 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:45:28.227 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.299 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:45:28.299 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:45:28.299 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.371 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:45:28.372 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:45:28.372 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.409 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:45:28.409 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:45:28.409 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.483 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:45:28.483 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:45:28.484 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.558 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:45:28.558 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:45:28.558 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.629 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:45:28.629 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:45:28.629 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:45:28.629 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.653 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:45:28.653 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:45:28.653 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.694 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:45:28.694 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:45:28.695 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:45:28.695 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.706 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:45:28.707 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:45:28.707 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.758 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:45:28.759 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:45:28.759 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:28.770 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:45:28.771 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:45:28.771 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:45:28.888 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:45:28.888 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:45:29.977 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:45:29.978 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:45:30.103 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:45:30.104 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:45:30.149 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:45:30.149 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:45:30.207 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:45:30.207 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:45:33.991 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:45:33.992 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:45:34.060 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:45:34.060 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:45:34.142 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:45:34.142 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:45:34.201 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:45:34.227 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/data/mon.a.tgz 2024-03-29T11:45:34.227 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-29T11:45:34.246 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.aGa8oM2MWU 2024-03-29T11:45:34.247 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.aGa8oM2MWU 2024-03-29T11:45:34.363 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.aGa8oM2MWU 2024-03-29T11:45:34.455 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.aGa8oM2MWU is 126KB 2024-03-29T11:45:34.700 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.aGa8oM2MWU 2024-03-29T11:45:34.716 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/data/mon.c.tgz 2024-03-29T11:45:34.716 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-29T11:45:34.772 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.o8EKbXLbdC 2024-03-29T11:45:34.773 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.o8EKbXLbdC 2024-03-29T11:45:34.864 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.o8EKbXLbdC 2024-03-29T11:45:34.953 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.o8EKbXLbdC is 126KB 2024-03-29T11:45:35.375 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.o8EKbXLbdC 2024-03-29T11:45:35.392 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/data/mon.b.tgz 2024-03-29T11:45:35.393 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-29T11:45:35.412 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.nGOTy6bPpj 2024-03-29T11:45:35.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.nGOTy6bPpj 2024-03-29T11:45:35.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.nGOTy6bPpj 2024-03-29T11:45:35.617 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.nGOTy6bPpj is 126KB 2024-03-29T11:45:35.815 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.nGOTy6bPpj 2024-03-29T11:45:35.832 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:45:35.833 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:45:35.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:45:35.922 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:45:35.963 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:45:35.963 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:45:36.115 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/remote/smithi022/crash 2024-03-29T11:45:36.154 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:45:36.226 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/remote/smithi204/crash 2024-03-29T11:45:36.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:45:36.277 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:45:36.277 DEBUG:teuthology.orchestra.run.smithi022:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:45:36.280 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:45:36.308 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41462.log 2024-03-29T11:45:36.309 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39782.log 2024-03-29T11:45:36.350 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41462.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:45:36.350 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39427.log 2024-03-29T11:45:36.350 INFO:teuthology.orchestra.run.smithi022.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41462.log.gz 2024-03-29T11:45:36.350 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-mon.a.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39782.log.gz 2024-03-29T11:45:36.350 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40135.log 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40135.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39427.log.gz 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39700.log 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41032.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40135.log.gz 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39310.log 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38998.log 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39310.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39700.log.gz 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39821.log 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39310.log.gz 2024-03-29T11:45:36.351 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41071.log 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38998.log.gz 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39821.log.gz 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38842.log 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41423.log 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41071.log.gz 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41423.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38842.log.gz 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41149.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41423.log.gz 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39037.log 2024-03-29T11:45:36.352 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40369.log/var/log/ceph/ceph-client.admin.39115.log: 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40369.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39037.log.gz 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38411.log 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40369.log.gz 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38294.log 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40174.log 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38294.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38411.log.gz 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38920.log 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38294.log.gz 2024-03-29T11:45:36.353 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39977.log 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39154.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38920.log.gz 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38372.log 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39977.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38372.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39977.log.gz 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39938.log 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38372.log.gz 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41618.log 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41188.log 2024-03-29T11:45:36.354 INFO:teuthology.orchestra.run.smithi022.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41618.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39938.log.gz 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41696.log 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41188.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41618.log.gz 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41227.log 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34970.log 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41188.log.gz 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41227.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41696.log.gz 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3%/var/log/ceph/ceph-client.admin.34970.log: -- replaced with /var/log/ceph/ceph-client.admin.41227.log.gz 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34651.log 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34970.log.gz 2024-03-29T11:45:36.355 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40486.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr: 86.2%/var/log/ceph/ceph-client.admin.34651.log: -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39661.log 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41384.log 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34651.log.gz 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40954.log 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41384.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39661.log.gz 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41501.log 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40954.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41384.log.gzgzip 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40447.log 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41501.log: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gz 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38686.log 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40447.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41501.log.gz 2024-03-29T11:45:36.356 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38764.log 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38686.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28844.log 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40447.log.gz 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38764.log: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.38686.log.gz 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38881.log 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph.tmp-client.admin.28844.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.38764.log.gz 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41306.log 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28844.log.gz 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40603.log 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40057.log 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38881.log.gz 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40603.log: gzip -5 --verbose -- 85.9% /var/log/ceph/ceph-client.admin.39544.log 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41306.log.gz 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-03-29T11:45:36.357 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40603.log.gz 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39544.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40057.log.gz 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35045.log 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39544.log.gz/var/log/ceph/ceph-client.admin.40642.log: 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.35045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40252.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35045.log.gz 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 86.4% 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38803.log 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38569.log 2024-03-29T11:45:36.358 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38803.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-03-29T11:45:36.359 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38333.log 2024-03-29T11:45:36.359 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38569.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38803.log.gz 2024-03-29T11:45:36.359 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39388.log 2024-03-29T11:45:36.359 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-03-29T11:45:36.359 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38569.log.gz 2024-03-29T11:45:36.359 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38333.log: /var/log/ceph/ceph-client.admin.39388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40096.log 2024-03-29T11:45:36.359 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40798.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38333.log.gz 2024-03-29T11:45:36.359 INFO:teuthology.orchestra.run.smithi022.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39388.log.gz 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41266.log 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40681.log 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40096.log: /var/log/ceph/ceph-client.admin.41266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39271.log 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40096.log.gz 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40681.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41266.log.gz 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38450.log 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40681.log.gz 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40759.log 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39271.log: /var/log/ceph/ceph-client.admin.38450.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39271.log.gz 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40759.log: gzip -5 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38450.log.gz 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38608.log 2024-03-29T11:45:36.360 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38959.log 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41345.log 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39583.log 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38608.log.gz 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5/var/log/ceph/ceph-client.admin.41345.log: --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38959.log.gz 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41345.log.gz 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39076.log: gzip -5 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39583.log.gz 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40017.log 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40408.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39232.log 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40330.log 2024-03-29T11:45:36.361 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40017.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40017.log.gz 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40330.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39232.log.gz 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39349.log 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40993.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41579.log 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39193.log 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41579.log: gzip -5 --verbose 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39349.log.gz 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr: -- /var/log/ceph/ceph-client.admin.39899.log 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39193.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41579.log.gz 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38647.log 2024-03-29T11:45:36.362 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39193.log.gz 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40213.log 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38647.log: gzip -5 --verbose 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39899.log.gz 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr: -- /var/log/ceph/ceph-client.admin.38530.log 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40213.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38647.log.gz 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38489.log 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38725.log 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38489.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39505.log 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38489.log.gz 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38725.log.gz 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.39505.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39505.log.gz 2024-03-29T11:45:36.363 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40564.log 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.2.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39860.log 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40291.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40564.log.gz 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39860.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39860.log.gz 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39466.log 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41657.log 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr: 93.8%/var/log/ceph/ceph-client.admin.39466.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:45:36.364 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-03-29T11:45:36.365 INFO:teuthology.orchestra.run.smithi022.stderr:gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39466.log.gz 2024-03-29T11:45:36.365 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2024-03-29T11:45:36.365 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41540.log 2024-03-29T11:45:36.365 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39739.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41657.log.gz 2024-03-29T11:45:36.365 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:45:36.365 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41540.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz 2024-03-29T11:45:36.365 INFO:teuthology.orchestra.run.smithi022.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41540.log.gz 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35174.log 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35094.log 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35014.log 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35174.log.gz 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35014.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35094.log.gz 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35294.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35014.log.gz 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:45:36.371 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35254.log 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.4.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35054.log 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.7.log: 87.7%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.35254.log.gz 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35214.log 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35054.log.gz 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35214.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mgr.x.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35214.log.gz 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:45:36.372 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35134.log 2024-03-29T11:45:36.373 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-client.admin.35134.log: 2024-03-29T11:45:36.373 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35134.log.gz 2024-03-29T11:45:36.425 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:45:48.332 INFO:teuthology.orchestra.run.smithi204.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:45:48.516 INFO:teuthology.orchestra.run.smithi204.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:45:48.597 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:45:49.871 INFO:teuthology.orchestra.run.smithi204.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:45:51.427 INFO:teuthology.orchestra.run.smithi204.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:45:52.671 INFO:teuthology.orchestra.run.smithi022.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:45:52.832 INFO:teuthology.orchestra.run.smithi022.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:45:54.353 INFO:teuthology.orchestra.run.smithi022.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:46:29.864 INFO:teuthology.orchestra.run.smithi022.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:46:31.169 INFO:teuthology.orchestra.run.smithi022.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:46:31.171 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:46:31.171 INFO:teuthology.orchestra.run.smithi022.stderr:real 0m54.876s 2024-03-29T11:46:31.171 INFO:teuthology.orchestra.run.smithi022.stderr:user 2m45.971s 2024-03-29T11:46:31.171 INFO:teuthology.orchestra.run.smithi022.stderr:sys 0m5.254s 2024-03-29T11:46:45.893 INFO:teuthology.orchestra.run.smithi204.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:46:45.895 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-29T11:46:45.895 INFO:teuthology.orchestra.run.smithi204.stderr:real 1m9.558s 2024-03-29T11:46:45.895 INFO:teuthology.orchestra.run.smithi204.stderr:user 1m57.743s 2024-03-29T11:46:45.895 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m4.358s 2024-03-29T11:46:45.895 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:46:45.896 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/remote/smithi022/log 2024-03-29T11:46:45.897 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:12.157 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/remote/smithi204/log 2024-03-29T11:47:12.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:31.025 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:31.277 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=46a4da64161c40eab4e0cf522396f328 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:31.281 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:47:31.290 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:47:31.298 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:47:31.299 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:47:31.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:47:31.391 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:47:31.402 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:47:31.402 DEBUG:teuthology.orchestra.run.smithi022:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:47:31.404 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:47:31.418 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:47:31.420 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:47:31.424 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:47:31.424 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-29T11:47:31.424 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 +396us[ +396us] +/- 73ms 2024-03-29T11:47:31.424 INFO:teuthology.orchestra.run.smithi022.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -114us[ -118us] +/- 68ms 2024-03-29T11:47:31.424 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -242us[ -245us] +/- 68ms 2024-03-29T11:47:31.424 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:47:31.426 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:47:31.426 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-29T11:47:31.426 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +397us[ +395us] +/- 73ms 2024-03-29T11:47:31.427 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -137us[ -139us] +/- 68ms 2024-03-29T11:47:31.428 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 -245us[ -245us] +/- 68ms 2024-03-29T11:47:31.428 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:47:31.428 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:47:31.438 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:47:31.439 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:47:31.447 DEBUG:teuthology.orchestra.run.smithi022:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:47:31.537 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:47:31.608 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:47:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:31.641 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:47:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:31.672 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:47:31.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:47:31.690 INFO:teuthology.task.internal:Duration was 1320.387607 seconds 2024-03-29T11:47:31.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:47:31.703 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:47:31.704 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:47:31.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:47:31.752 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:47:31.757 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:47:32.131 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:47:32.131 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:47:32.131 DEBUG:teuthology.orchestra.run.smithi022:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:47:32.155 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-03-29T11:47:32.156 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:47:32.196 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:47:32.196 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:47:32.201 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:47:32.274 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:47:32.275 DEBUG:teuthology.orchestra.run.smithi022:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:47:32.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:47:32.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:47:32.444 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:47:32.444 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:47:32.476 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:47:32.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:47:32.516 DEBUG:teuthology.orchestra.run.smithi022:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:32.520 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:32.544 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = core 2024-03-29T11:47:32.570 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-03-29T11:47:32.594 DEBUG:teuthology.orchestra.run.smithi022:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:32.631 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:47:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:47:32.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:47:32.662 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:47:32.663 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/remote/smithi022 2024-03-29T11:47:32.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:47:32.708 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717/remote/smithi204 2024-03-29T11:47:32.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:47:32.746 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:47:32.747 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:47:32.750 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:47:32.797 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:47:32.812 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:47:32.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:47:32.825 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:47:32.825 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:47:32.829 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:47:32.843 INFO:teuthology.orchestra.run.smithi022.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:47 /home/ubuntu/cephtest 2024-03-29T11:47:32.852 INFO:teuthology.orchestra.run.smithi204.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:47 /home/ubuntu/cephtest 2024-03-29T11:47:32.853 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:47:32.864 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:47:32.885 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:47:32.901 DEBUG:teuthology.nuke:shortname: smithi022 2024-03-29T11:47:32.902 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:47:32.924 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:39.636671', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQSUy3yjxAe3hzCysAxj6mi6w9pBA8hIMXdJW134/u3hWyuyWtQiYzvUWmAkLB5se5uJTsiT3NHlNnPnP7UCFE='} 2024-03-29T11:47:32.929 DEBUG:teuthology.nuke:shortname: smithi204 2024-03-29T11:47:32.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:47:32.949 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:39.637900', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvWN01mvFk7+LYeo9d5kf4S4xDRmeshlRoq70u6g/fyxryJ0WCJ9fmGzz1gk3ng0Bf1y7+XgNUOixyAHcUjHUU='} 2024-03-29T11:47:32.983 INFO:teuthology.orchestra.console.smithi022:Power off 2024-03-29T11:47:32.983 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:47:33.001 INFO:teuthology.orchestra.console.smithi204:Power off 2024-03-29T11:47:33.001 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:47:33.014 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2024-03-29T11:47:33.014 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:33.028 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-03-29T11:47:33.028 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:33.051 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-29T11:47:33.053 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-29T11:47:37.053 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:37.061 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:37.163 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-29T11:47:37.175 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-29T11:47:41.166 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:41.176 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:47:41.277 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-29T11:47:41.286 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-03-29T11:47:41.378 INFO:teuthology.orchestra.console.smithi022:Power off completed 2024-03-29T11:47:41.387 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-03-29T11:47:41.491 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1320.3876070976257 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=46a4da64161c40eab4e0cf522396f328 status: fail success: false 2024-03-29T11:47:41.492 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:47:41.586 INFO:teuthology.run:FAIL